ramesh padmanabha's slides on mercury interactive loadrunner
TRANSCRIPT
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 11
Performance Load TestingPerformance Load TestingCase Study – Agilent Case Study – Agilent
TechnologiesTechnologies
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 22
AgendaAgenda
IntroductionsIntroductions BackgroundBackground Testing ObjectivesTesting Objectives Preparation PhasePreparation Phase Execution PhaseExecution Phase Analysis Analysis Lessons LearntLessons Learnt Contact InformationContact Information
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 33
IntroductionIntroduction
Ramesh PadmanabhanRamesh Padmanabhan Entegration SoftwareEntegration Software Consulting & product company based Consulting & product company based
in San Josein San Jose Proud to be service partners ofProud to be service partners of
Oracle CorporationOracle Corporation
Mercury InteractiveMercury InteractiveYash TechnologiesYash Technologies
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 44
IntroductionIntroduction
Agilent TechnologiesAgilent Technologies $6 Billion Global Mfg Company$6 Billion Global Mfg Company Over 30,000 employees in more than Over 30,000 employees in more than
50 countries50 countries One of the largest global single One of the largest global single
instance installs of Oracle E-business instance installs of Oracle E-business suitesuite
Consolidated over 150 legacy systemsConsolidated over 150 legacy systems Expect a maximum 5,000 concurrent Expect a maximum 5,000 concurrent
usersusers
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 55
BackgroundBackground
Largest single instance installLargest single instance install 3 HP Superdomes –Production, 3 HP Superdomes –Production,
Reporting, PlanningReporting, Planning Single US based data centerSingle US based data center Over 50 operating unitsOver 50 operating units Significant business volume in Significant business volume in
Asia & EuropeAsia & Europe Consolidating over 125 Consolidating over 125
different legacy systemsdifferent legacy systems Implemented all Financial & Implemented all Financial &
MFG ModulesMFG Modules
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 66
Testing ObjectivesTesting Objectives
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 77
Testing ObjectivesTesting Objectives
Validate single instance strategyValidate single instance strategy Validate network and hardware Validate network and hardware
infrastructureinfrastructure Scalability to 5000 concurrent usersScalability to 5000 concurrent users Stress test for “high water mark”Stress test for “high water mark” Set user response time expectationsSet user response time expectations Identify and fix significant performance Identify and fix significant performance
tuning issues within Oracle Applicationstuning issues within Oracle Applications Identify and drive solutions for hardware Identify and drive solutions for hardware
issuesissues
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 88
Preparation PhasePreparation Phase
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 99
Data GatheringData Gathering
Identified major transactions within each Identified major transactions within each application moduleapplication module
Questionnaires sent for legacy data Questionnaires sent for legacy data volumes by geography (US, Asia, Europe)volumes by geography (US, Asia, Europe)
Short listed transactions with high volume Short listed transactions with high volume or data intensive processingor data intensive processing
Identified user distribution by region and Identified user distribution by region and by application areas by application areas
Determined estimation methodology for Determined estimation methodology for inquiry transactions inquiry transactions
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1010
Hardware PreparationHardware Preparation
Ensure that the production configuration of Ensure that the production configuration of back-end server and middle tier machines back-end server and middle tier machines were set-up and configuredwere set-up and configured
Procure the Load generation agent boxes Procure the Load generation agent boxes and have them installed and setup at the and have them installed and setup at the right locations right locations
Ensure that the Cisco load balancing router Ensure that the Cisco load balancing router was correctly set upwas correctly set up
Set up network ‘sniffing’ devices to get Set up network ‘sniffing’ devices to get detailed metrics of network traffic detailed metrics of network traffic
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1111
Software PreparationSoftware Preparation
Procure and install LoadRunner on the Procure and install LoadRunner on the agent and controller boxesagent and controller boxes
Install LoadRunner and the Oracle Install LoadRunner and the Oracle Applications client on the machines of Applications client on the machines of the scriptersthe scripters
Install/Setup other database Install/Setup other database monitoring softwaremonitoring software
Prepare scripts for detailed Prepare scripts for detailed transaction analysistransaction analysis
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1212
Data PreparationData Preparation
Validated various application Validated various application setups setups
Initial cycles required all key Initial cycles required all key master data to be fabricatedmaster data to be fabricated
Developed numerous scripts to Developed numerous scripts to extract key data elements like extract key data elements like items, customers, vendors etc. to items, customers, vendors etc. to be used in transactionsbe used in transactions
Ensured adequate breadth of data.Ensured adequate breadth of data. Identified key data and parameters Identified key data and parameters
for background loadfor background load
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1313
Develop LoadRunner ScriptsDevelop LoadRunner Scripts
Recorded scripts for all the critical and Recorded scripts for all the critical and high volume transactionshigh volume transactions
Adequate mix of inquiry and update Adequate mix of inquiry and update txns.txns.
Parameterized all the critical pieces of Parameterized all the critical pieces of data like item, customer, orders etc.data like item, customer, orders etc.
Identified activities for which server Identified activities for which server response times were key and set up response times were key and set up transaction timers around them e.g. transaction timers around them e.g. commits, quick-picks etc.commits, quick-picks etc.
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1414
Execution PhaseExecution Phase
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1515
Build Test ScenariosBuild Test Scenarios
Develop matrix for users by Develop matrix for users by geography by transactiongeography by transaction
Manual scenariosManual scenarios Goal oriented scenariosGoal oriented scenarios Transactions split into three Transactions split into three
groups based on data groups based on data dependency conditionsdependency conditions
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1616
Run Tests…Run Tests…
5 cycles of testing5 cycles of testing 1- validation cycle1- validation cycle 2 – complete cycle with converted data2 – complete cycle with converted data 3- Stress test cycle3- Stress test cycle 4- Complete integrated test with key 4- Complete integrated test with key
interfaces and customizationsinterfaces and customizations 5- Production simulation run5- Production simulation run
Each cycle comprised of two major Each cycle comprised of two major runs/day for two weeks. Each test runs/day for two weeks. Each test run was about 4-7hrs longrun was about 4-7hrs long
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1717
Run Tests…Run Tests…
5000 concurrent user load 5000 concurrent user load generated from 8 LoadRunner generated from 8 LoadRunner agents – 4 in US, 2 each in agents – 4 in US, 2 each in Europe & AsiaEurope & Asia
LoadRunner monitors set up for LoadRunner monitors set up for network, backend server & network, backend server & middle-tier boxesmiddle-tier boxes
Dedicated DBA and performance Dedicated DBA and performance tuning experts monitored the HP tuning experts monitored the HP Superdome serverSuperdome server
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1818
Analysis Analysis
Used LoadRunner Analysis toolUsed LoadRunner Analysis tool Real time graphical interface to Real time graphical interface to
monitor the test progressmonitor the test progress Post run analysis includes numerous Post run analysis includes numerous
graphs and transaction timersgraphs and transaction timers More detailed analysis was done More detailed analysis was done
from the result data stored by from the result data stored by LoadRunner in an Access databaseLoadRunner in an Access database
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 1919
AnalysisAnalysis
Data from the analysis used toData from the analysis used to Set up realistic response time Set up realistic response time
expectations from the end usersexpectations from the end users Modify various database parameters Modify various database parameters
in the init.ora to better performancein the init.ora to better performance Tweak settings of the Cisco load Tweak settings of the Cisco load
balancer for middle tier machinesbalancer for middle tier machines Identify and tune some of the Identify and tune some of the
application code that had bad application code that had bad performanceperformance
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 2020
LimitationsLimitations
Some performance intensive processes Some performance intensive processes could not be tested due to data could not be tested due to data dependency issues e.g. lock-box receiptsdependency issues e.g. lock-box receipts
Some dynamic and interactive processes Some dynamic and interactive processes could not be tested very well e.g. could not be tested very well e.g. configured ordersconfigured orders
Some custom code not stable till the last Some custom code not stable till the last cyclecycle
Some of the newer application modules Some of the newer application modules not stable for a reasonable testnot stable for a reasonable test
Application version and patch set lagsApplication version and patch set lags
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 2121
Lessons LearntLessons Learnt
Performance test will only be as Performance test will only be as good as the data collected in the good as the data collected in the analysis phaseanalysis phase
While performance test can While performance test can significantly reduce risk of poor significantly reduce risk of poor performance, it is not a guarantyperformance, it is not a guaranty
Initial performance testing cycles Initial performance testing cycles should focus more on non-code should focus more on non-code related performance variables related performance variables
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 2222
Lessons LearntLessons Learnt
Intensive code related performance testing Intensive code related performance testing & tuning should take place after custom & tuning should take place after custom solutions have been put into testing and solutions have been put into testing and application patch sets are frozenapplication patch sets are frozen
Performance testing should be in the Performance testing should be in the critical path of project plan and critical path of project plan and performance testing instances should be performance testing instances should be patched just like the BST instancespatched just like the BST instances
Should plan on at least one marathon Should plan on at least one marathon testing run that extends for 3 or 4 daystesting run that extends for 3 or 4 days
April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 2323
Contact Information
Ramesh PadmanabhanEntegration Softwarerpadmanabhan@entegration.com408-674-3701www.entegration.com