benchmark factory user guide 6-8-1
DESCRIPTION
Benchmark Factory is an intuitive, easy to use database load testing tool. The following links provide you with information to quickly start the testing process and gather resultsTRANSCRIPT
-
Benchmark Factory forDatabases 6.8.1
User Guide
-
Copyright
2012 Quest Software, Inc.
ALL RIGHTS RESERVED.
This guide contains proprietary information protected by copyright. The software described inthis guide is furnished under a software license or nondisclosure agreement. This software may beused or copied only in accordance with the terms of the applicable agreement. No part of thisguide may be reproduced or transmitted in any form or by any means, electronic or mechanical,including photocopying and recording for any purpose other than the purchasers personal usewithout the written permission of Quest Software, Inc.
The information in this document is provided in connection with Quest products. No license,express or implied, by estoppel or otherwise, to any intellectual property right is granted by thisdocument or in connection with the sale of Quest products. EXCEPT AS SET FORTH INQUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FORTHIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMSANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTSINCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT,INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES(INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESSINTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE ORINABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THEPOSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respectto the accuracy or completeness of the contents of this document and reserves the right to makechanges to specifications and product descriptions at any time without notice. Quest does notmake any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact:
Quest Software World HeadquartersLEGAL Dept5 Polaris WayAliso Viejo, CA 92656email: [email protected]
Refer to our Web site (www.quest.com) for regional and international office information.
Patents
Toad for Cloud Databases contains patent pending technology.
Trademarks
-
Benchmark Factory for Databases User GuideCopyright
3
Quest, Quest Software, the Quest Software logo, Benchmark Factory, Spotlight, LiteSpeed,Simplicity at Work, Toad, TOAD, T.O.A.D., and TOADWORLD and vToadare trademarks andregistered trademarks of Quest Software, Inc. For a complete list of Quest Softwares trademarks,see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registeredtrademarks used in this guide are property of their respective owners.
Benchmark Factory for Databases
User Guide
September 2012
-
Table of Contents
Copyright 2
Getting Started 17
Getting Started 17
Benchmark Factory Community 17
17
New in this Release 17
New for Release 6.8.1 17
New for Release 6.8 18
New For Release 6.7 18
Understanding Benchmark Factory 20
About Benchmark Factory 20
The Benchmark Factory Environment 20
Integration with Other Quest Software Products 21
Understanding the Benchmark Factory Environment 21
Understanding Benchmark Factory Terminology 22
How Benchmark Factory Works 25
Create a Script With Benchmark Factory 26
1. Set Up Benchmark Factory 27
Benchmark Factory Components 35
BenchmarkFactory Console 35
Agent(s) 36
Run Reports 37
Cutting and Pasting of Run Reports 38
Testing Server Throughput 38
Testing SQL Scalability 39
Testing Real Application Clusters 40
-
Benchmark Factory for Databases User GuideTable of Contents
5
Repository Manager 41
Views 44
Script View 44
Jobs Queue View 45
Properties View 47
Agents View 47
Output View 47
Properties 47
How Do I... 91
Overview 91
How do I... 91
User Scenario Wizard 92
Profile Creation Wizard 92
Overview 92
Edit a Profile 94
Submit Job Dialog 94
Run Reports 96
Cutting and Pasting of Run Reports 97
Testing Server Throughput 97
Testing SQL Scalability 98
Testing Real Application Clusters 99
Advanced Creation Objects (Object Mapping) 100
Benchmark Objects Wizard 101
Changing Scaling Factors 103
To Change Scale Factors 103
Execute File Wizard 105
BFScript Wizard 105
-
Benchmark Factory for Databases User GuideTable of Contents
6
Script Wizard 105
Script Assist 107
SQL Scalability Wizard 107
Overview 107
Overview Tab 109
109
Transactions 109
Real-Time 110
Summary 110
Messages 111
Copy a Load Scenario to a Replay, Mix, Goal, or Scalability Test 112
Running SQL Preview 113
Replace Child Latencies 113
Goal Testing Quick Start 115
Overview 115
Actions Perfomred in This Quick Start 115
Use Global Search and Replace 124
Creating an ODBC Trace File 125
Real Application Clustering (RAC) Quick Start 125
Overview 125
Purpose of this Quick Start 126
Actions Performed in This Quick Start 126
Save the Selected Job to the Script View 136
Schedule A Job 136
About Latency 137
Set the Number of Users on an Agent 138
Stagger User Startup 140
-
Benchmark Factory for Databases User GuideTable of Contents
7
Running Benchmark Factory with Multiple Agents 140
Benchmark Factory Console 140
Benchmark Factory Agent 141
Running a Job With Multiple Agents 141
Creating Load Scenarios 142
Load Scenario Wizard 142
Overview 142
Settings 144
About Settings 144
General Settings 144
Benchmarks Settings 146
Timing Settings 147
User Load (Measurement Intervals) 148
LatencySettings 150
Repository Settings 152
Agent Settings 154
Profiles 156
Overview 156
Edit Profiles 156
Drivers 158
Profile Properties 163
Overview 163
General Tab 163
Options Tab 165
Profile Properties Real-Time Counters Tab 166
Clustering Tab 169
-
Benchmark Factory for Databases User GuideTable of Contents
8
Wizards 171
Overview 171
Benchmark Objects Wizard 171
Benchmark Objects Wizard 171
Data Repository Migration Wizard 173
Data Repository Migration Wizard 173
Execute File Wizard 174
Execute File Wizard 174
Load Scenario Wizard 175
Load Scenario Wizard 175
Capture and Replay Oracle Workload 176
Industry Standard Benchmark (TPC-C, TPC-H...) 192
Validate Oracle RAC Cluster Scalability 193
Test Database Scalability 195
Test SQL For Scalability 196
Replay Load from an Oracle Trace File 197
Test Stored Procedures for Scalability 200
Replay Load from SQL Server Trace Table 201
Import from Foglight Performance Analysis Repository 202
Import from a Delimited Text File 203
Replay Load From an ODBC Trace File 205
Import from Oracle Dynamic Performance View 206
Import from Quest Tools Export Files 207
Create a Custom Load Scenario (Scalability, Goal...) 208
Mix Test 208
Replay Test 209
Goal Test 210
Scalability Test 212
-
Benchmark Factory for Databases User GuideTable of Contents
9
Profile Creation Wizard 213
Profile Creation Wizard 213
BFScript 214
BFScript Wizard 214
SQL Scalability Wizard 217
SQL Scalability Wizard 217
Submit Job Dialog 222
Submit Job Dialog 222
Transaction Wizard 224
Transaction Wizard 224
Test Stored Procedures for Scalability 225
Create SQL Statement Transaction 226
Create Prepared SQL Statement Transaction(s) 227
Create Execute Prepared SQL Statement Transaction 231
Import Transactions 232
User Scenario Wizard 233
User Scenario Wizard 233
Import and Replay Load Scenarios From an Oracle Trace File 233
Replay Load from a SQL Server Trace Table 234
Import from a Delimited Text File 235
Replay Load from an ODBC Trace File 235
Create a Custom User Scenario 236
Testing Results (Run Reports) 237
Benchmark Factory TestingResults 237
Benchmark Factory Testing Results Terminology 237
Exporting Test Results to Excel Using the Command Line 238
Real-Time Load Testing Results 239
Overview-Viewing Real-time Statistics Workloads 239
-
Benchmark Factory for Databases User GuideTable of Contents
10
Change Graph Views 239
Summary Tab 242
Realtime Statistics Tab 242
Run vs.Userload Tab 244
Transaction Node 246
Benchmark Factory Run Reports 248
Run Reports 248
Change Graph Views 251
Run Reports Viewer 254
View Load Testing Results 256
Using Benchmark Factory Run Reports 257
View Multiple Run Results 270
Export BenchmarkFactory Test Results to Excel 274
Benchmarks 276
Overview 276
Realistic Expectations When Using Benchmarks 276
What Benchmarks Measure 276
Provided Benchmarks 276
AS3AP Benchmark 277
Best Practices 277
Scalable Hardware Benchmark 278
How the Scalable Hardware Benchmark Works 278
Scaling Factor 278
Best Practices 278
TPC-B Benchmark 279
Overview 279
Certification of Transaction Processing Council (TPC) Testing Results 279
Best Practices 279
-
Benchmark Factory for Databases User GuideTable of Contents
11
Scaling Factor 280
TPC-C Benchmark 280
Overview 281
TPC-C Tables 281
Certification of Transaction Processing Council (TPC) Testing Results 282
Best Practices 282
TPC-D Benchmark 283
Overview 283
Certification of Transaction Processing Council (TPC) Testing Results 283
Best Practices 283
Scaling Factor 284
TPC-E Benchmark 284
Overview 284
Certification of Transaction Processing Council (TPC) Testing Results 285
Best Practices 285
Scaling Factor 285
TPC-H Benchmark 285
Overview 285
Certification of Transaction Processing Council Testing Results 286
Best Practices 286
Scaling Factor 286
BFScripts 288
About Scripts 288
Using Scripts in Username and Password Fields 290
Dates and Numbers 290
$BFCreditCardExp 290
$BFCurrentDate 290
BFCurrentDateTime 291
-
Benchmark Factory for Databases User GuideTable of Contents
12
$BFDate 292
File Access 293
$BFFileArray 293
Global Variable 294
$BFGetGlobalVar 294
$BFSetGlobalVar 294
$BFSetGlobalVarRtn 295
Name and Address 295
$BFAddress 295
$BFAddress2 295
$BFCity 296
$BFCompany 296
$BFCountry 297
$BFEmail 297
$BFFirstName 297
$BFFullName 298
$BFLastName 298
$BFMiddleInitial 298
$BFPhone 299
$BFState 299
$BFZipCode 300
Numerical Manipulation 300
$BFFormat 300
$BFProd 301
$BFSum 301
Random Numbers 302
$BFRand 302
$BFRandRange 302
-
Benchmark Factory for Databases User GuideTable of Contents
13
$BFURandRange 303
$BFURand 303
Random String 304
$BFCreditCard 304
$BFRandList 304
$BFRandMultiList 305
$BFRandStr 306
$BFURandList 307
$BFList 307
String Manipulation 308
$BFAsc 308
$BFChr 308
$BFConcat 309
$BFLeft 309
$BFLen 310
$BFLower 310
$BFMid 311
$BFRight 311
$BFTrim 311
$BFTrimLeft 312
$BFTrimRight 312
$BFUpper 313
Test Info 313
$BFGetVar 313
$BFMaxNode 314
$BFNode 314
$BFNumberOfIterations 314
$BFProfile 315
-
Benchmark Factory for Databases User GuideTable of Contents
14
$BFRunID 315
$BFSetVar 316
$BFSetVarRtn 316
$BFUserCounter 317
$BFUserID 318
$BFUserLoad 318
$BFNextUserload 318
$BFPrevUserload 319
Quick Starts 320
About the Quickstarts 320
Real Application Clustering (RAC) Quick Start 320
Overview 320
Purpose of this Quick Start 321
Actions Performed in This Quick Start 321
Upgrading Your Database Server Quick Start 331
Actions Performed in This Quick Start 331
Oracle Trace Import-Database Replay Quick Start 339
Overview 339
Actions Performed in This Quick Start 339
Standard Benchmark Quick Start 346
Overview 346
Actions Performed Duirng this Quick Start 346
Goal Testing Quick Start 353
Overview 353
Actions Perfomred in This Quick Start 354
Troubleshooting 363
Agent Connection 363
Use Benchmark Factory with SQL Server 2005 Client 363
-
Benchmark Factory for Databases User GuideTable of Contents
15
MySQL Initialization Settings 363
Appendix 364
Change Graph Views 364
Show Data/Show Graph 364
Graph Legend 365
Toolbar 366
Print 366
Copy Data to Clipboard 366
Copy Graph to Clipboard 366
Load Configuration 366
Save Configuration 366
Set as Default 366
Clear Configuration 367
Customize List Controls 367
Create a SQL 7 Trace Table Using the SQL Server Profiler 367
Create a SQL 2000 Trace Table Using the SQL Server Profiler 368
Create a SQL 2005 Trace Table Using the SQL Server Profiler 369
Oracle Instant Client Installation 370
Migrating Repository Data Using the DOS Command Line 371
Store Procedure Examples 372
Oracle Trace File Activation 373
Support Bundle 375
System Requirements/Upgrade Requirements/Supported Databases 376
System Requirements 376
Shortcut Keys 378
Copyright, Licensing, and Licensing Agreement 380
Copyright 380
-
Benchmark Factory for Databases User GuideTable of Contents
16
Licensing and Virtual User Packs 381
Modify the Benchmark Factory License 382
Modify the Benchmark Factory Virtual User Pack 383
Legal Notice 383
384
384
Appendix:Contact Quest 396
Contact Quest/About Quest Software 396
About Quest 396
Contacting Quest Software 396
ContactingQuest Support 396
Index 398
-
Getting Started
Getting StartedBenchmark Factory is an intuitive, easy to use database load testing tool. The following linksprovide you with information to quickly start the testing process and gather results.
l Video: Take a Tour of Benchmark Factoryl About Benchmark Factory (page 20)
l Understanding the Benchmark Factory Environment (page 21)
l Understanding Benchmark Factory Terminology (page 22)
l How Benchmark Factory Works (page 25)
l Benchmark Factory Components
l Create a Script With Benchmark Factory (page 26)
l Benchmark Factory TestingResults (page 237)
l Quick Starts
l Real Application Clustering (RAC) Quick Start (page 320)
l Upgrading Your Database Server Quick Start (page 331)
l Oracle Trace Import-Database Replay Quick Start (page 339)
l Standard Benchmark Quick Start (page 346)
l Goal Testing Quick Start (page 353)
l Quest Benchmark Factory Web Site (contains additional videos and white papers)
Benchmark Factory Community
Get the latest product information, find helpful resources, and join a discussion with theBenchmark Factory for Databases team and other community members. Join the BenchmarkFactory for Databases community at http://www.benchmarkfactory.inside.quest.com.
New in this Release
New for Release 6.8.1
This release of Benchmark Factory for Databases provides bug fixes.
1
-
Benchmark Factory for Databases User GuideGetting Started
18
New for Release 6.8
New features in this release:
l New Oracle Workload Capture Method-This release of Benchmark Factory forDatabases provides a new method to accomplish Oracle Capture/Replay using Oracle'sFine Grained Auditing (FGA) objects.
l Customer Improvement Program -Benchmark Factory now tracks feature utilization inorder to provide feedback to improve Benchmark Factory for Databases.
l Automatic Partitioning-Automatic Partitioning of TPC-H tables and indexes for Oracletesting has been added.
l User Interface for SQLStatements-The user interface for SQLStatements has beenenhanced.
l Reports- A performance comparison report has been added to Run History to compareOracle runs.
l Wizards-The Oracle and SQL Server Capture wizards user interfaces have been streamlinefor easier use.
New For Release 6.7
New features in this release:
l Improved Run Reports User Interface-Benchmark Factory Run Reports has beenimproved to provide easier access to test results tables and graphs.
-
Benchmark Factory for Databases User GuideGetting Started
19
l Import and Export of Test Results-You can now import and export test results formRun Reports.
l Increased Benchmark Loading Speed-Benchmarks used in the database testing processnow load faster to reduce database testing time.
-
Understanding Benchmark Factory
About Benchmark FactoryBenchmark Factory is a database performance and code scalability testing tool that simulatesusers and transactions on the database and replays production workload in non-productionenvironments. This enables developers, DBAs, and QA teams to validate that their databases willscale as user load increases, application changes are made, and platform changes areimplemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL, andother databases via ODBC connectivity. The following topics provides an overview of theBenchmark Factory Environment:
l The Benchmark Factory Environment (page 20)
l Understanding the Benchmark Factory Environment (page 21)
l Understanding Benchmark Factory Terminology (page 22)
l How Benchmark Factory Works (page 25)
l Create a Script With Benchmark Factory (page 26)
The Benchmark Factory EnvironmentBenchmark Factory places enormous stress on a database system, which is typically hard toachieve in a standard testing environment. A system typically breaks under extreme load. Byidentifying system capacity and performance bottlenecks before they occur, Benchmark Factoryfacilitates proactive testing, which in turn reduces downtime, development costs, and potentialloss of revenue. Benchmark Factory allows you to:
l Determine system throughput and capacity for database systems
l Simulate thousands of concurrent users with a minimal amount of hardware
l Find applications that do not scale well with an increase in the number of users
l Find breaking points, weak links, or bottlenecks of a system
l Quantify application or server performance
All test results are collected and stored in the repository for data analysis and reporting.Benchmark Factory collects a vast amount of statistics, including overall server throughput(measured in transactions per second, bytes transferred, etc.) and detailed transaction statistics by
2
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
21
individual workstation producing a load. You use these statistics to measure, analyze, andpredict the capacity of a system.
Integration with Other Quest Software Products
Benchmark Factory integrates with other Quest Software products to allow IT departments toquickly measure the capacity and performance of their systems to ensure that users willexperience fast response times. For instance, Benchmark Factory can run multiple workloads inconjunction with Spotlight products. This provides the ability to detect and diagnose issues,allowing users to resolve bottlenecks, slow performance, and application flaws before anapplication is entered into production.
Understanding the Benchmark Factory EnvironmentNote: The Benchmark Factory Quick Starts provide procedures that introduce you to theBenchmark Factory environment. See "About the Quickstarts" (page 320) for more information.
The following provides a narrative on the terminology, application, and mechanics of usingBenchmark Factory.
Automated Testing, Virtual Users, and Agents
When you load test with Benchmark Factory, you are performing automated testing. Automatedtesting is the process of using software to test computer hardware or software implying that thesoftware is used instead of actual users. The software acts as a "virtual" user. For example, sayyou would like to test your database with two hundred virtual users over a given period of time.Benchmark Factory lets you select two hundred virtual users and the length of time you wish toperform the test.
Benchmark Factory uses Agents to deploy the virtual users. An Agent is a software routine thatwaits in the background and performs an action when a specified event occurs. One Agent cansimulate thousands of virtual users (limited by hardware and workload characteristics) at a time.Each virtual user has their own connection to the system under test.
Understanding Benchmarks
A benchmark is a performance test of hardware and/or software on a system-under-test.Benchmark Factory provides the option of using industry standard benchmarks during the loadtesting process. Benchmarks measure system peak performance when performing typicaloperations.
Benchmark Factory comes equipped with the following industry standard benchmarks:
l AS3AP Benchmark (page 277)
l Scalable Hardware Benchmark (page 278)
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
22
l TPC-B Benchmark (page 279)
l TPC-C Benchmark (page 280)
l TPC-D Benchmark (page 283)
l TPC-E Benchmark (page 284)
l TPC-H Benchmark (page 285)
Understanding Benchmark Factory TerminologyThe following provides a list of terminology required to understand the Benchmark Factory loadtesting process.
l A script is the user's workspace that allows you to create and save jobs.
l Scripts are created in the Script View. See "Script View" (page 44) for moreinformation.
l When ready to run, jobs are submitted to the Jobs Queue View. See "Jobs QueueView" (page 45) for more information.
l The Load Scenario Wizard is the starting point for the creation of scripts. See"Load Scenario Wizard" (page 175) for more information.
l A job contains load scenario, execution of a file, creation/deletion of benchmark objects,or a combination of each.
l A load scenario is an assembled test comprised of user scenarios and/or transactions.These tests can be run with multiple virtual users. A load scenario can be one of thefollowing:
l Mixed Workload: A Mixed Workload test runs for a specified time at eachpredetermined user load level. Each user will run a transaction mix based uponthe weights defined on the transactions tab. For example, if a test has twotransactions, A and B, with A having a weight of one and B having a weight offour, on average B will run four times for every time A is run once. The run orderwill be randomly generated for each user so they are not all running the sametransaction simultaneously. That run order is used for that user each time the test isperformed to ensure reproducible results.
l Goal Test: A goal test is used to find maximum throughput or response timegoals. A transaction mix is executed at user load levels, determined by setting abeginning, ending, and interval value. When run, the specified goal criterion isevaluated at the end of each iteration and the test ends once the goal or maximumuser load has been reached.
l Replay Test: A Replay Test runs multiple transactions with each one runningindependently on a specified number of users. The test will run until the definednumber of executions for each transaction or a specified time limit is reached.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
23
l SQL Scalability Test: A SQL Scalability test executes each transactionindividually for each userload and timing or execution period. For example, atest could have two transactions, A and B, and two userloads of 10 and 20, withan iteration length of one minute. Transaction A would execute continually forone minute at userload 10, then B would do the same. Next A will run at userload20, followed again by test B, for a total time of 4 minutes.
Click here for load scenario creation procedures.
l A user scenario is a series of one or more transactions to be executed in order to make asingle transaction, normally associated with user behavior simulated against the system-under-test. An Execute SQL Statement from a file User Scenario will execute SQLstatements as defined in an XML file. See "User Scenario Wizard " (page 233) for moreinformation.
l A transaction is a single unit of work in the testing process, used in load scenarios anduser scenarios. For example, a SQL statement. Click here for procedures to create atransaction.
l Execute a file allows you to execute a file during the running of a job. See "Execute FileWizard" (page 174) for more information.
l Create or delete benchmark objects configures a system-under-test for industry standardbenchmark tests to measure system performance. Each standard benchmark has beendeveloped with specific system configuration requirements, that include tables, indexes,data, etc. See "Benchmark Objects Wizard" (page 171) for more information.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
24
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
25
How Benchmark Factory WorksThe following steps provide an overview of how Benchmark Factory components interact duringthe load testing process.
1. The BenchmarkFactory Console (page 35) implements the workload testing process andcontrols one or more distributed agent machines. Each agent machine can simulatethousands of users. Each simulated user executes transactions and records statistics.
2. Benchmark Factory Agent(s) (page 36)machines simulate virtual users. The Agents sendtransactions to the system-under-test. The Agents record statistics that includes how muchdata the resulting transaction contained and how long it took to get the results. At theend of an iteration, each agent machine reports its findings back to Benchmark Factory.
3. A server (system-under-test) is the database Benchmark Factory connects to. BenchmarkFactory is server neutral and network protocol independent. Benchmark Factory usesvendor client libraries of the system-under-test. Any system that the client software cansupport, Benchmark Factory can support. Its only requirement is that Agent machinesmust be able to connect to a server through an appropriate manner and it supports thetest system.
4. The Repository Manager (page 41) stores all testing data.
5. Results are viewed from Run Reports (page 248).
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
26
Create a Script With Benchmark FactoryThe following provides a narrative discussion on how to create a script using BenchmarkFactory. This process consists of following steps:
1. Set up Benchmark Factory
2. Create a Script
3. Create a Profile
4. Create a Job
5. Run the Script
6. Analyze Test Results
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
27
Note: The Benchmark Factory Quick Starts provide procedures that introduce you to theBenchmark Factory environment. See "About the Quickstarts" (page 320) for more information.
1. Set Up Benchmark Factory
The Settings window displays default values used when creating new scripts. Changes to thesesettings affect only new load scenarios, not existing load scenarios.
To Display Benchmark Factory Settings
l Select Edit | Settings. The settings dialog displays.
The Settings window provides the following tabs:
l General Settings (page 144)
l Benchmarks Settings (page 146)
l Timing Settings (page 147)
l LatencySettings (page 150)
l Repository Settings (page 152)
l Agent Settings
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
28
2. Create a Script
Using the Benchmark Factory graphical user interface, you can create scripts in the Script View.You open one script at a time when using Benchmark Factory. See "BenchmarkFactory Console"(page 35) for more information.
All tasks required to complete a Benchmark Factory script can be accomplished using wizards.The first wizard you will use is the Load Scenario Wizard. The Load Scenario Wizard takes youthrough the steps required to create a script and complete a database load test. See "LoadScenario Wizard" (page 175) for more information.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
29
The first step the Load Scenario wizard asks you to perform is to select a load scenario type fromthe load scenario dialog. Load Scenarios are an assembled load test comprised of:
l User scenarios: A series of one or more transactions to be executed in order normallyassociated with a behavior to be tested on the system-under-test. These are thecomponents used to build a load scenario.
l Transaction: A single unit of work in the testing process, such as retrieving a Web page,executing a SQL statement, writing a file, or sending an email.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
30
After selecting the desired task type, the Load Scenario Wizard will guide you through the stepsfor script creation, depending on which type of task you select.
3. Create a Profile
Profiles are sets of information that Benchmark Factory uses to communicate with the system-under-test . This information includes, server name, IP address, and user name. You createprofiles using the:
l Profile Creation Wizard: (See "Profile Creation Wizard" (page 213) for more information.
l Edit Profiles (See "Edit a Profile" (page 94) for more information.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
31
4. Create a Job
A job holds a load scenario, execution of a file, creation/deletion of benchmark objects, or acombination of each. Jobs are created in Benchmark Factory when a load scenario is added to ascript. A script can hold as many jobs as you chose to create.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
32
Change Script Properties
Benchmark Factory script properties can be changed or adjusted to meet load testing requirement.You change properties by clicking on the node of the property you wish to change. For example,if you wish to change the user load test you click on the Load Scenario node. See About theScript Node for more information.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
33
To change the User Load properties, select the User Load tab and select the desirednumber of users.
5. Run the Script
After completing the wizards and setting in the desired properties, you run the job by selecting
the Job Node and clicking the Submit Job icon , or right-clicking and selecting Submit Job.See "Submit Job Dialog " (page 222) for more information.
-
Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory
34
6. Analyze Test Results
Benchmark Factory provides a number of metrics to analyze a system-under-test. These include:
l Response, retrieval, and transactions times
l Transaction throughput in transactions per second (TPS)
l Amounts of data transferred in bites per second (BPS)
l Errors incurred
l Transaction Distribution
l Any user defined counters (Windows NT/2000 Performance Monitor) added using theBenchmark Factory Console.
Benchmark Factory provides the following testing results:
l Real time statistics (See "Realtime Statistics Tab " (page 242) for more information.)
l Run Reports (See "Run Reports " (page 248) for more information.)
l Testing results exported to Excel (See "Export BenchmarkFactory Test Results to Excel"(page 274) for more information.)
-
Benchmark Factory Components
BenchmarkFactory ConsoleThe Benchmark Factory console implements the database workload testing process. Thisinterface is where:
l Scripts are created and initiated. A script is the user workspace that allows you to designand save jobs.
l Jobs are created and submitted to the Jobs Queue View. See "Jobs Queue View" (page45) for more information.
l All script properties are set. See About the Script Node for more information.
l Settings can be made. See "About Settings" (page 144) for more information.AboutScripts (page 288)
l Users can access test results. See "Benchmark Factory TestingResults" (page 237) formore information.
l Agents are managed. See "Agent(s)" (page 36) for more information.
The Benchmark Factory Console contains the following views:
l Script View (page 44)
l Jobs Queue View (page 45)
l Properties View (page 47)
l Agents View (page 47)
l Output View (page 47)
3
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
36
Agent(s)Benchmark Factory Agent(s) reside on each physical client machine and spawn multiple virtual-user sessions. Each Agentgenerates an information screen with tabs that contain a variety of Agent information. Benchmark Factory can controlhundreds of Agent machines. Each Agent can simulate up to 1500 virtual users, depending upon hardware and testconfiguration. Virtual users simulate the load against the system-under-test. Each virtual user is a separate thread, actingindependently of the other virtual users, with its own connection to the system-under-test. Each virtual user tracks its ownstatistics that includes transaction times and the number of times a transaction executes.
The Agent provides the follow tabs:
l Agent Test Status Tab
l Agent Virtual Users Tab
l Agent Transaction Tab
l Agent Output Tab
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
37
Run ReportsBenchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.
Note: Three instances of Run Reports can be viewed at one time.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
38
Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. SeeCut and Pasting Reports for more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you toview a variety of server throughput statistics that include:
l Transaction/Second (TPS)
l Transaction Time
l Bytes/Second (BPS)
l Rows/Second (RPS)
l Total Bytes
l Total Errors
l Total Rows
l Response Time
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
39
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of theindividual transaction performance as they compare to each other. This is most useful whenanalyzing a SQL Scalability test.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
40
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determinesystem capacity. The cluster performance graph allows users to view the performance ofindividual nodes.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
41
Repository ManagerNote: If you create a new Benchmark Factory 5.5 repository, earlier versions of BenchmarkFactory will not work against this repository.
The Repository is a database where all of the test results are stored. Benchmark Factory insertstest results into the repository and provides an easy way to access the data. By default, theRepository is a MySQL database that resides on the same machine as Benchmark Factory forDatabases. The Repository can reside on another database server if required. Tto change thedatabase select the Data Source Name of the ODBC connection for the new database.
Note: If the database structure does not exist on the selected database, a prompt to create thestructure will appear when OK is clicked.
The supported repository database types are:
l Sybase Adaptive Server Anywhere
l Microsoft SQL Server
l MySQL
l Oracle
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
42
To launch the Repository Manager
1. Select Start | All Programs | Quest Software | Benchmark Factory for Databases |Repository | Repository Manager. The Repository Manager dialog displays.
Section Field/Icon Description
ConnectionParameters
Data SourceName
Data Source name of the ODBCconnection used to connect to therepository database.
ODBC Driver Current ODBC driver
User Name The User Name used to log into theselected database.
Password The Password associated with the username used to log into the database.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
43
Section Field/Icon Description
Displays the ODBC connectioninformation dialog for the selected datasource.
Displays the ODBC Data SourceAdministrator dialog, used to add andedit ODBC connections.
Tests the connection of the currentlyselected ODBC Data Source.
Maintenance Creates the repository objects on theselected database.
Deletes the repository objects on theselected database.Warning: This will delete all testresults stored in the Repository.
Launches the Data Migration Wizard.See "Data Repository MigrationWizard" (page 173) for moreinformation.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
44
Views
Script View
The Script View displays the currently open script and provides user workspace that allows youto design and save jobs. When ready to run, jobs are submitted to the Jobs Queue. The ScriptView Toolbar provides user functionality when creating or editing load testing scripts.
Note: You must be in the script view to create jobs and edit jobs.
Script View Toolbar
The Script View toolbar provides user functionality when creating or editing load testing scripts.
Creates a blank job. You must click on the Script Node to active the icon.
Displays the Load Scenario Wizard (page 175)
Displays the User Scenario Wizard (page 233)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
45
Displays the Transaction Wizard (page 224)
Displays the Benchmark Objects Wizard (page 171)
Displays the Execute File Wizard (page 174)
Displays the Submit Job Dialog (page 222)
Moves an item up in the Script View
Moves an item down in the Script View
Jobs Queue View
From the Jobs Queue, jobs are run, status can be viewed, and real-time testing results can beviewed. The Jobs Queue Toolbar (page 46) provides user functionality to save or delete jobs.The following job statues are provided:
l Scheduled: All jobs currently waiting to run or are scheduled to run a future time.
l Running: Job currently running.
l Completed: All completed jobs.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
46
Jobs Queue Toolbar
Jobs Queue icons provides user functionality when running jobs.
Deletes the selected job from the Jobs Queue.
Saves the selected job to the Script View. See "Save the Selected Job to the Script View"(page 136) for more information.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
47
Properties View
The Properties View allows you to monitor and managed scripts. Properties can beviewed for the:
l Script Node Properties (page 48)
l Jobs Node
Agents View
The Agent view displays the status of connected agents.
Output View
The Output window displays errors and messages that occur outside of running a job. Forexample, a .dll error or re-registering a benchmark.
Properties
About the Properties View
Benchmark Factory properties allow you to:
l Adjust script parameters
l View script properties
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
48
l Access Benchmark Factory Run Reports See "Run Reports " (page 248) for moreinformation.
l View realtime statistics (See "Realtime Statistics Tab " (page 242) for more information.
l Set and view latency See "About Latency " (page 137) for more information.
The following provides Benchmark Factory properties definitions:
l Script (See About the Script Node for more information.)
l Job (See Jobs Node for more information.)
l Latency (See "About Latency " (page 137) for more information.)
l Benchmark Properties See "Benchmark Properties" (page 87) for more information.)
Script Node Properties
Script Node
The Script node displays the Benchmark Factory Home page and Help options.
Jobs Node
Note: In the Jobs Queue view, not all properties are editable.
The Jobs node provides the following tabs:
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
49
l General
l Alerts
l Job Schedule
l Real-Time Counters
l Global Variables Node (page 72)
Load Scenario Node
Note: In the Jobs Queue view, no properties are editable.
The Load Scenario node provides the following tabs:
l Transaction Mix Tab (page 49)
l Transaction Tab
l Goal Test (page 210)
l User Load Tab (MeasJobs Nodeurement Intervals)
l Timing Tab
l Options Tab-Mixed Workload, Goal, and Scalability Test (page 54)
Transaction Mix Tab
Note: The Transaction Mix tab is only editable in the Script View.
This tab allows you to run a transaction mix based on the assigned weight. If a test hastwo transactions, A and B, with A having the weight of one and B having the weight ofsix, transaction B will run six for every time A runs once. The run order is randomlygenerated for users.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
50
Settings Description
Name Name assigned to the transaction.
Weight Weight assigned to a transaction in a mix.
Percentage Percentage assigned to the transaction.
TotalWeight
Total weight of all the combined transactions.
Deletes a selected transaction from the mix.
User Scenario Node
The User Scenario node properties allows you to:
l Delete transactions
l Adjust transaction order
l Adjust latency (See "About Latency " (page 137) for more information.)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
51
User Scenario From File
The User Scenario From File node properties allows you to:
l Executing a SQL Statements from a File (page 51)
l Adjust latency (See "About Latency " (page 137) for more information.)
Executing a SQL Statements from a File
When large Oracle trace files are imported into Benchmark Factory, they are converted toeditable XML files.
To Edit SQL Statements from a file
1. Click Edit.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
52
2. Click File | Open.
The Open dialog displays.
3. Navigate to and select the required file XML file.
4. Click Open.
5. Under Session Details, click on the SQL statement you wish to edit. That statementdisplays in the SQL Statement view.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
53
6. From the Prepare SQL tab, you can click in the upper right-hand corner to:
l Run SQL Preview See "Running SQL Preview" (page 113) for moreinformation.
l Launch the BFScript Wizard See "BFScript Wizard" (page 214) for moreinformation.
7. After you finish making the required edits, click File| Save to save your edits.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
54
Transaction Node
Note: In the Jobs Queue view, no properties are editable.
The transaction node displays properties based on transaction type.
To view the Transaction node
Click the Transactions node in the Jobs Queue or Script View.
Options Tab-Mixed Workload, Goal, and Scalability Test
The Options tab specifies Repository, error handling, file execution, and databasecheckpoint settings.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
55
Section Field/Icon Description
Repository Settings Save Real-Time CounterInformationNote: To change real-timecounters please refer to ProfileProperties Real-Time CounterTab.
Saves counters andperformance monitorcounters to the Repositoryduring test. Sample rate isdetermined in job settings.
Save User Statistics Saves user level statistics
Save Response Distribution Saves transaction persecond and response timeper user distribution.
Log Transaction Results to disk Logs the transaction resultsof a job to disk.Note: Checking this optionimposes performancepenalties.
Error Handling Stop test after first error When checked, the test isstopped when an error is
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
56
Section Field/Icon Description
reported.
Execute the followingprogram at thebeginning of eachiteration
File NameNote: This field acceptsBFScripts.
Name of the program toexecute at the beginning ofan iteration.
Enforce Timeout Enforces a timeout on thefile being executed.
Execute the followingprogram at the end ofeach iteration
File NameNote: This field acceptsBFScripts.
Name of the program toexecute at the end of aniteration.
Enforce Timeout Enforces a timeout on thefile being executed so thatif the program does notcomplete in the specifiedtime, it will be stopped, andthe load scenario willcontinue.
Database Checkpoints Perform checkpoint at start ofeach test iteration
Initiates a databasecheckpoint at the beginningof a test iteration.
Perform checkpoints during eachiteration
Initiates a databasecheckpoint during a testiteration.
Number of checkpoints Specifies the number ofcheckpoints to initiate.
Options Tab-Replay Test
The Repository Properties tab controls what information is saved in the Repository during therunning of a test, and allows you to set error handling properties. The database where allBenchmark Factory information is saved. It contains information about the transactions stored ineach benchmark DLL and is where all of the statistical information is stored about benchmarkexecutions.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
57
To change Repository properties
1. Click the Load Scenario node.
2. Click the Options tab.
Setting Description
Repository Settings
Save Real-Time CounterInformation
Saves Benchmark Factory real time counters andperformance monitor counters to the repository duringtest.
Save User Statistics Saves user level statistics.
Save Distribution Saves transactions per second and response timedistribution.
Error Handling Stops the test when an error is returned.
Run Reports
Benchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
58
With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.
Note: Three instances of Run Reports can be viewed at one time.
Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. SeeCut and Pasting Reports for more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you toview a variety of server throughput statistics that include:
l Transaction/Second (TPS)
l Transaction Time
l Bytes/Second (BPS)
l Rows/Second (RPS)
l Total Bytes
l Total Errors
l Total Rows
l Response Time
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
59
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of theindividual transaction performance as they compare to each other. This is most useful whenanalyzing a SQL Scalability test.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
60
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determinesystem capacity. The cluster performance graph allows users to view the performance ofindividual nodes.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
61
SQL Statement Node
The SQL statement node displays individual SQL statements and allows you edit a SQLstatement or use the BFScript Wizard. See "BFScript Wizard" (page 214) for more information.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
62
Create Objects Node
Note: In the Jobs Queue view, no properties are editable.
This node allows you to either create the benchmark objects or delete the benchmark objects.Benchmark objects are used by standard benchmark load scenarios. Benchmarks can be added toa job using the Benchmark Objects Wizard. See "Benchmark Objects Wizard" (page 171) formore information.
Field Description
Benchmark Name of the benchmark whose benchmark properties are beingedited.
Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmark selected.For example, the AS3AP scales by a factor of 10 with valid scales of"1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1,with valid scales of "1, 2, 3, 4..."
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
63
Field Description
Displays the Benchmark Properties page and allows you to edit thebenchmark scale and map database tables and indexes. See"Benchmark Properties" (page 87) for more information.
Always deleteobjects beforecreate
Checking this box deletes benchmark objects before creating newones.When unchecked, Benchmark Factory checks for the benchmarktables. If they exist, then the creation will be skipped.
Note: The scale factor is not checked.
Distribute loadusing BenchmarkFactory Agents
Distributes data generation across multiple agent machines to speedup the standard benchmark data load.
Number of virtualusers to performcreation/load
Number of virtual users used to perform the test.
Execute File Node
Note: In the Jobs Queue view, no properties are editable.
The Execute File node allows you to place a file execution in a job as a sibling to a loadscenario. The file executes during the running of a job.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
64
Field Description
Execute File Type in or Select the File Name to execute:
Name of the file you wish to execute.
Note: This field accepts BFScripts. See "About Scripts" (page288) for more information.
Click to open a file.
Enforce Timeout Places a time constraint on the number of seconds a file has toexecute.
Enter a numberbetween 30 and 10000[n] Secs.
The number of seconds a file has to execute.
Jobs Queue Node Properties
Using the Jobs Node and Tabs
Note: Some of the nodes in the Jobs Queue are the same as the Script view. Not all nodes in theJobs Queue are editable and provide property viewing only.
The Jobs Node and tabs allow you to edit or view job parameters. These tabs can be usedwhen jobs are:
l Scheduled (page 64)
l Running (page 65)
l Completed (page 66)
Scheduled
When a job is scheduled to run, it is placed in the Scheduled Jobs Queue.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
65
Note: Not all of the tabs are editable in the schedule section.
The Job Queue provides the following tabs:
l Job Setup Tab (page 66)
l Create Objects Node (page 67)
l Run Job Tab (page 68)
l See Result Tab (page 69)
Running
As a job runs, clicking on the Run Job tab allows you to view realtime testing results.
The following provides details on realtime testing viewing:
l Summary Tab (page 242)
l Realtime Statistics Tab (page 242)
l Run vs.Userload Tab (page 244)
l Transaction vs. Load Tab (page 82)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
66
Completed
Jobs in the Completed Jobs Queue section allow you to perform limited test edits and completedtesting results using the See Result Tab (page 69).
Note: Not all of the tabs are editable in the schedule section.
The following provides tab details in the Completed tests section:
l Job Setup Tab (page 66)
l Create Objects Node (page 67)
l Run Job Tab (page 68)
l See Result Tab (page 69)
Job Setup Tab
This tab allows to view or edit job parameters and provides the following tabs:
l General
l Jobs Alerts
l Real-Time Counters
l Global Variables Node (page 72)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
67
Create Objects Node
Note: In the Jobs Queue view, no properties are editable.
This node allows you to either create the benchmark objects or delete the benchmark objects.Benchmark objects are used by standard benchmark load scenarios. Benchmarks can be added toa job using the Benchmark Objects Wizard. See "Benchmark Objects Wizard" (page 171) formore information.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
68
Field Description
Benchmark Name of the benchmark whose benchmark properties are beingedited.
Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmark selected.For example, the AS3AP scales by a factor of 10 with valid scales of"1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1,with valid scales of "1, 2, 3, 4..."
Displays the Benchmark Properties page and allows you to edit thebenchmark scale and map database tables and indexes. See"Benchmark Properties" (page 87) for more information.
Always deleteobjects beforecreate
Checking this box deletes benchmark objects before creating newones.When unchecked, Benchmark Factory checks for the benchmarktables. If they exist, then the creation will be skipped.
Note: The scale factor is not checked.
Distribute loadusing BenchmarkFactory Agents
Distributes data generation across multiple agent machines to speedup the standard benchmark data load.
Number of virtualusers to performcreation/load
Number of virtual users used to perform the test.
Run Job Tab
This RunJob tab allows you to schedule jobs and view job results. It provides thefollowing tabs:
l Schedule
l Summary Tab (page 242)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
69
l Realtime Statistics Tab (page 242)
l Run vs.Userload Tab (page 244)
l Transaction vs. Load Tab (page 82)
See Result Tab
The see results tab allows you to:
l Select results of tests based on profile selection
l Sort test results by successful completion or those that completed with errors.
l View graphs of completed tests.
To use the See Results Tab:
1. From the Profiles drop-down, select the All Profiles or the required profile.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
70
2. From the Status drop-down, select the required run type.
3. Click on the required requied test to view graphs.
4. To delete a test, select the test and click Delete in the upper right section of the dialog.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
71
5. To view Run Reports, select the required test and click the Show Test Results button inthe upper right section of the dialog.
See "Run Reports " (page 248) for more information.
Load Scenario Node
Note: In the Jobs Queue view, no properties are editable.
The Load Scenario node provides the following tabs:
l Transaction Mix Tab (page 49)
l Transaction Tab
l Goal Test (page 210)
l User Load Tab (MeasJobs Nodeurement Intervals)
l Timing Tab
l Options Tab-Mixed Workload, Goal, and Scalability Test (page 54)
User Scenario Node
The User Scenario node properties allows you to:
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
72
l Delete transactions
l Adjust transaction order
l Adjust latency (See "About Latency " (page 137) for more information.)
Transaction Node
Note: In the Jobs Queue view, no properties are editable.
The transaction node displays properties based on transaction type.
To view the Transaction node
Click the Transactions node in the Jobs Queue or Script View.
Global Variables Node
From the global variable node, you can add or delete the required variables.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
73
To add a global variable:
1. Right-click to display the global variable dialog.
2. Enter the name and value.
3. Click Add.
The new global variable dislays.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
74
Run Reports
Benchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.
Note: Three instances of Run Reports can be viewed at one time.
Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. SeeCut and Pasting Reports for more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you toview a variety of server throughput statistics that include:
l Transaction/Second (TPS)
l Transaction Time
l Bytes/Second (BPS)
l Rows/Second (RPS)
l Total Bytes
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
75
l Total Errors
l Total Rows
l Response Time
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of theindividual transaction performance as they compare to each other. This is most useful whenanalyzing a SQL Scalability test.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
76
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determinesystem capacity. The cluster performance graph allows users to view the performance ofindividual nodes.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
77
Summary Tab
The Summary Tab properties displays information on the job when it is running. You canmonitor the status of the test, which userload its running, transactions per second (TPS), and avariety of other datapoints that give you insight into the actual performance your databaseprovides under the userload being tested.
Iteration Overruns
Iteration overruns occur at the end of an iteration to allow time for all transactions submittedwithin the test iteration cycle to complete, so that all transaction statistics can be collected. Forexample, an agent may execute a transaction during the last five seconds of test iteration, if thistransaction takes 15 seconds to complete, an iteration overrun of 10 seconds will occur.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
78
Realtime Statistics Tab
The Real Time Statistics node provides you with real-time graphs and raw data. This data allowsyou to spot system-under-test issues that may be affecting server performance. The Real TimeStatistics node displays real-time counter data. Right-clicking inside the graph displays a drop-down that allows you to change graph settings and view.
To view Real-Time Statistic
1. In the Jobs Queue, click the job running.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
79
2. Select Click here to Add Datapoints. The Add Datapoints dialog displays.
3. Select the desired datapoints to view.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
80
3. Click OK. The graph displays with the selected datapoints. (To change graph views,right-click. See "Change Graph Views" (page 364) for more information.)
Run vs.Userload Tab
The Run vs. Load tab compares the user load transaction against the selected datapoint allowingyou to view how the various datapoints are being affected by the job running.
To view Run Vs.Userload tab testing results:
1. While the job is running, click the RunJob tab, then select the Run vs.Userload tab.
2. Click the datapoints Vs. Userload drop-down and select the desired datapoint.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
81
The graph updates with the selected datapoint.
3. To change graph views, right-click. See Change Graph Views for more information.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
82
Transaction vs. Load Tab
The Transaction vs. Load tab compares the user load transaction against the selected datapointallowing you to view how the various datapoints are being affected by the job running.
To view Transaction vs.Load tab testing results:
1. While the job is running, click the RunJob tab, then select the Run vs.Userload tab.
2. Click the datapoints vs. Userload drop-down and select the required datapoint.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
83
The graph updates with the selected datapoint.
3. To change graph views, right-click. See "Change Graph Views" (page 364) for moreinformation.
Messages Node
The Messages node displays:
l Job status
l Any errors or warnings
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
84
Latency
About Latency
Latency is the speed in which transactions are sent to a server or the period between receivingresults of one transaction and sending the next transaction to a serve. Benchmark Factory allowsyou to select the following latency models and distribution models:
Latency Types Distribution Models
No Delay (page 84) Absolute Distribution (page 86)
Interarrival Time (page85)
Uniform Distribution (page 86)
Keying Time (page 85) Negative Exponential Distribution (page86)
Think Time (page 85) Normal Distribution (page 86)
Poisson Distribution (page 87)
No Delay
No Delay means that transactions execute as fast as possible. As soon as one transaction isprocessed, the next transaction is issued against the server. In the case of a mixed workload test,each virtual user issues transactions as fast as possible. The No Delay option is used when thegoal of the test is to stress the system to its limits, without concern for accurately simulating
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
85
users. With No Delay specified, a relatively low number of users can stress the system to itslimits. As such, there is no easy way to correlate N virtual users running with no delay to somenumber of real users.
Interarrival Time
Interarrival Time derives its name from achieving a specific transaction rate as seen by a server(the rate at which transactions are arriving at a server.) When specifying Interarrival Time,Benchmark Factory is instructed, that regardless of how long a transaction actually takes toexecute, to ensure that the transactions arrive at a server at a specific interval.
Keying Time
Keying Time is used to simulate an amount of time performing data entry, before executing atransaction (entering information). In many cases, Keying Time is used with Think Time toprovide a delay before and after a transaction execute. Selecting Keying Time inserts a fixed orvariable delay before executing a transaction.
Think Time
Think Time is used to simulate an amount of time "to think" about the results of a previoustransaction. This could be time spent performing analysis on the results of a database query.Selecting Think Time inserts a fixed or variable delay after executing a transaction.
Latency Distribution Models
About
Benchmark Factory provides the following distribution models
l Absolute Distribution (page 86)
l Uniform Distribution (page 86)
l Negative Exponential Distribution (page 86)
l Normal Distribution (page 86)
l Poisson Distribution (page 87)
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
86
Absolute Distribution
Absolute is a fixed length delay. If a 2000 ms delay is specified for Keying Time (page 85) and a3000 ms delay is specified forThink Time (page 85), each time the transaction should execute,Benchmark Factory waits 2000 ms, then fires the transaction, then waits an additional 3000 msbefore deciding which transaction to execute next. If an Inter-Arrival model is chosen with adelay type of Absolute set to 2000 ms, Benchmark Factory marks the time, executes thetransaction, and assuming the transaction finishes in less than two seconds, waits until twoseconds from the marked time has elapsed before determining which transaction to execute next.
Uniform Distribution
Selecting a Uniform delay instructs that random delay should be chosen, with an equalprobability of being the minimum value, the maximum value, or any value in between. Uniformdelays are chosen when it is suspected that the delay is highly random within a range or aminimal amount of statistical analysis has been performed to determine how the actual usersreact. Suppose a uniform Keying Time (page 85) is selected with a minimum value of 1000 msand a maximum value of 1500 ms. If the transaction is executed more than 500 times, there is ahigh probability that each possible delay has been selected at least once. With the other delaytypes, this is not the case. If the same 2000 ms to 2500 ms uniform delay is set for Interarrivaltime, the tester essentially is setting the test so that a server sees the transaction every 2 to 2.5seconds, instead of exactly two seconds as in the Absolute delay time.
Negative Exponential Distribution
Similar to the Normal distribution, Negative Exponential inserts a random delay based on amathematical model. A Negative Exponential distribution should be chosen when most usershave latency of the mean latency or less. Negative Exponential is weighted heavily towardsmaller latencies being selected relative to the mean and maximum latency.
Normal Distribution
Normal distributions differ from Uniform delays in that most of the delays chosen by BenchmarkFactory will be close to the average, but can vary by as much as 10% of the mean. While aUniform delay is used when users have latencies within equal likelihood of being anywherebetween two values, Normal distributions are chosen when all users fall within a range, but mostof the modeled users have a latency close to the average latency.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
87
Poisson Distribution
A Poisson distribution is very similar to the Normal distribution and can be used most placeswhere a Normal distribution delay could be used. The biggest difference between a Normaldistribution and a Poisson distribution is that Poisson selects discreet values.
Benchmark Properties
Benchmark Properties
Benchmark scale factors are used to increase the size of the database testing tables during thecreation of benchmark objects, and to correctly execute the benchmark transactions as needed tofully utilize the database objects created.
The Benchmark Properties page displays information on:
l Scales (page 87)Scales (page 87)
l Advanced (page 89)
l History Tables (page 89)
Scales
The scale dialog specifies the benchmark scale used to create table data
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
88
Field Description
Benchmark Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmarkselected. For example, the AS3AP benchmark scales by a factor of10 with valid scales of "1, 10, 20, 30..." and the TPC-C benchmarkscales by a factory of 1, with valid scales of "1, 2, 3, 4..."
Show EmptyTables
Displays any tables that will be created but not populated withdata.
Estimated Size The approximate size that the tables will use in the databasewithout indexes. Additional space is needed for indexes.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
89
Advanced
Advancer mapping provides views of the tables and indexes that allows you to determine wherethere tables and indexes will be stored in the database. This allows your database to run moreefficiently. See "Advanced Creation Objects (Object Mapping)" (page 100) for more information.
History Tables
History tables allows you to set the number of history tables to create. The TPC-B benchmark ismade up of only one transaction that updates three tables and inserts a record into a history table.Inserting one record into one history table limits testing performance. The Benchmark Factoryproperties page allows the user to set the number of history tables to create during a test. Thebest ratio of history tables to virtual users is based on database configuration and hardware. Thenumber of history tables to use is determined by the tester.
-
Benchmark Factory for Databases User GuideBenchmark Factory Components
90
Field Description
Number of Historytables
Allows you to set the number of history tables to create.
-
How Do I...
OverviewThe How Do I section points you to the procedures required to perform load testing.
How do I...l Create a Load Scenario - This wizard walks you through the steps to create and run
scripts and load scenarios.
l Create a User Scenario - This wizard walks you through the steps to create auser scenario.
l Create a Profile - This wizard walks you through the steps to hook up to the data baseyou are testing.
l Edit a Profile
l Submit a Job - This wizard walks you through the steps to submit a job to the JobsQueue for testing.
l Use Run Reports
l Use Advanced Creation Objects
l Create Benchmark Objects for Load Testing This wizard walks you through the steps tocreate benchmark objects for load testing.
l Execute a File - This wizard walks you through the steps to execute a file outside ofBenchmark Factory.
l Change Scale Factors
l Use Benchmark Factory Scripting Capabilities - This wizard walks you through the stepsto use Benchmark Factory built-in functions for use in scripts and the load testing process.
l Use SQL Scalability Wizard
l Copy a Load Scenario to a Replay Test, Mix Test, or Goal Test
l Run SQL Preview
l Replace Child Latencies
l Perform a Goal Test
l Use Global Search and Replace
l Create a Customized ODBC Trace File
l Run a Real Application Cluster (RAC) Workload
l Save the Selected Job to the Script View
4
-
Benchmark Factory for Databases User GuideHow Do I...
92
l Set Latency
l Set the Number of Users on an Agent
l Stagger User Startup
l Run Benchmark Factory with Multiple Agents
User Scenario WizardThe User Scenario Wizard allows you to build user scenarios. A user scenario is a series of oneor more transactions to be executed in order, normally associated with user behavior simulatedagainst the system-under-test. From the User Scenario Wizard you can:
l Import and Replay Load Scenarios From an Oracle Trace File
l Replay load from SQL Server trace table
l Import from a delimited text file
l Replay from an ODBC trace File
l Create a custom user scenario
Profile Creation Wizard
Overview
Profiles are used by Benchmark Factory to connect to the system-under-test. The profile containsinformation required to connect to the system-under-test, such as server name, IP address, anduser name. The Profile Creation Wizard walks you through steps required to create a BenchmarkFactory profile.
To create a profile
1. Select Wizards | Profile Creation.
2. Click Next.
3. From the Driver drop-down list, select the required driver type.
4. Click Next.
5. Enter the User/Schema.
-
Benchmark Factory for Databases User GuideHow Do I...
93
6. Enter the Password.
7. From the TNSDriver drop-down, select the required database.
or
Click the Direct tab and enter the required fields.
8. From the Connect as drop, select the required connection, Normal, SYSDBA, orSYSOPER.
9. Select the installed client if applicable.
10. Click the Verify connection before proceeding check box to verify the connection to thespecified database.
11. Click Next.
12. Enter a profile name.
13. Click Finish.
-
Benchmark Factory for Databases User GuideHow Do I...
94
Edit a ProfileTo edit a profile
1. Click the Edit Profile icon .
2. Double-click on a profile to edit or click View Properties in the upper right corner ofthe dialog.
The Edit Profiles dialog displays. The following options are provided.
l Adds a profile
l Deletes a profile
l Displays profile properties allowing you to edit the profile
Submit Job DialogA job contains load scenario, execute file, creation/deletion of benchmark objects, or acombination of each. The Submit Job Wizard walks you through the process of adding a job tothe jobs queue. See "Jobs Queue View" (page 45) for more information.
To submit a job to the Jobs Queue
1. Select the Job node.
2. Right-click.
3. Select Submit Job. The Submit Job dialog displays.
-
Benchmark Factory for Databases User GuideHow Do I...
95
The Submit Job dialog allows you to tailor your job needs and provides thefollowing options:
Section Description
Profile Allows you to select profiles already created.
Displays the Profile Creation Wizard. See"Profile Creation Wizard" (page 213) for moreinformation.
Displays the Profile setting dialog. See"General Tab" (page 163) for more information.
Actions Allows you to:
Run Job: Submits the job to the queue and runs thejob.
Run Job when queue becomes idle: Submits the jobto the jobs queue and puts it in a ready state so thatit will run when the currently ready or running jobcompletes.
-
Benchmark Factory for Databases User GuideHow Do I...
96
Section Description
Run Job per schedule: Submits the next jobscheduled.
Start Agent and Run Job: Starts the agent and runsthe job.
Start Agent and Run Job when queue becomesidle: Starts the agent and runs the job when allprevious jobs have finished.
Start Agent and Run Job per schedule: Starts theagent and runs the next job scheduled.
No action: Closes the dialog after you selectSubmit.
Comments Allows you to add a comment to the job.
Alerts Allows you to send an email, pager, or net-sendoperator alerts when the job completes.
Job Counters Allows you to add Job Counters to the job.
Stop job afterfirst error
When checked, the job will stop if an error isencountered.
4. After filling in the desired parameters, click Submit.
Run ReportsBenchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.
Note: Three instances of Run Reports can be viewed at one time.
-
Benchmark Factory for Databases User GuideHow Do I...
97
Cutting and Pasting of Run Reports
From Benchmark Factory run reports you can cut and paste test results into the Clipboard. SeeCut and Pasting Reports for more information.
Testing Server Throughput
Server throughput is the measurement of database capacity. Benchmark Factory allows you toview a variety of server throughput statistics that include:
l Transaction/Second (TPS)
l Transaction Time
l Bytes/Second (BPS)
l Rows/Second (RPS)
l Total Bytes
l Total Errors
l Total Rows
l Response Time
-
Benchmark Factory for Databases User GuideHow Do I...
98
Testing SQL Scalability
The Benchmark Factory transaction comparison graph provides users with a comparison of theindividual transaction performance as they compare to each other. This is most useful whenanalyzing a SQL Scalability test.
-
Benchmark Factory for Databases User GuideHow Do I...
99
Testing Real Application Clusters
Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determinesystem capacity. The cluster performance graph allows users to view the performance ofindividual nodes.
-
Benchmark Factory for Databases User GuideHow Do I...
100
Advanced Creation Objects (Object Mapping)The Advanced Creation Option allows you to set advanced properties for tablespaces anddatafiles, used for the creation of standard benchmark tables and indexes.
To create advanced objects
1. From the Database Type drop-down, select the desired database type.
2. Select the Tables or Indexes radio button.
3. Select the object(s) you wish to map.
-
Benchmark Factory for Databases User GuideHow Do I...
101
4. Click Edit Options. The Mapping Options dialog displays.
5. Select the desired profile from the Load Tablespaces from Profile drop-down.
6. Select the desired options.
7. Click OK.
8. To save the mapping options, click Save. The Save Options File As dialog display.
9. Enter the desired name.
10. Click Save.
Benchmark Objects WizardThe Benchmark Objects Wizard creates or clears the database objects (tables/indexes, etc.)needed to run industry standard benchmarks. The Benchmark Objects Wizard is able to run ifyou have a root node job, load scenario, Create/Delete Benchmark object, or Execute File nodeselected. The Create/Delete Benchmark object node will be added just below the item selected oras a child of the job.
The Benchmark Objects Wizard:
l Selects the benchmark to create objects for
l Determines the scale of a benchmark
l Creates an instruction to create the benchmark objects
To create a Benchmark Object
Note: To create, delete, and modify database tables/indexes you must have the user loginrights required.
1. Select the Job or Load Scenario node.
2. Select Wizards | Benchmark Objects.
-
Benchmark Factory for Databases User GuideHow Do I...
102
3. Click Next.
4. From the drop-down, select the desired benchmark.
5. Click Next.
6. Select the desired benchmark scale in Benchmark Scale. When setting up a load test,Benchmark Factory allows you to change the scaling factor of the standard benchmarksprovided. Scales factors increase the size of a database during the testing process allowingrealistic testing. By the scale factors, the number of rows added to the tables increases,allowing for larger userloads to be used, placing a greater stress on the system under test.During load testing, you can run various test with numerous scaling factors to ensure yourdatabase performs at various performance levels.
7. The Advanced Creation Option allows you to set advanced properties for tablespaces anddatafiles, used for the creation of standard benchmark tables and indexes. To add mappingoptions click Advanced Creation Option or proceed to step 18.
8. From the Database Type drop-down, select the desired database type.
9. Select the Tables or Indexes radio button.
10. Select the object(s) you wish to map.
11. Click Edit Options. The Mapping Options dialog displays.
12. Select the desired options.
13. Click OK.
14. To save the mapping options, click Save. The Save Options File As dialog display.
15. Enter the desired name.
16. Click Save.
17. Click OK.
18. Click Next. The Create/Delete benchmark objects dialog displays and provides thefollowing options:
l Create Benchmark Objects: Creates and initializes benchmark objects (tables, indexes,etc.) required to run the standard benchmark transactions.
-
Benchmark Factory for Databases User GuideHow Do I...
103
l Always delete objects before create: Deletes existing benchmark objects (tables,indexes, etc.) before creating new ones.
Note: Checking the "Always delete objects before create" check box will delete allbenchmark objects in the selected database and then re-create the objects in thedatabases. When the checkbox is unchecked, Benchmark Factory checks for thebenchmark tables. If they exist, then the creation will be skipped. The scale factor isnot checked.
l Delete Benchmark Objects: Deletes existing benchmark objects (tables, indexes, etc.).
19. Select the desired option.
20. Click Next. The Benchmark Objects Wizard completion dialog displays.
21. Click Finish. The Benchmark Object node creates and displays in the Script View.
Changing Scaling FactorsWhen setting up a load test, Benchmark Factory allows you to change the scaling factor of thestandard benchmarks provided. Scales factors increase the size of a database during the testingprocess allowing realistic testing. By changing the scale factors, the number of rows added to thetables increases, allowing for larger userloads to be used, placing a greater stress on the system-under-test. During load testing, you can run various test with numerous scaling factors to ensureyour database performs at varying user load levels.
Benchmark Factory provides the following scalable benchmarks:
l AS3AP
l Scalable Hardware
l TPC-B
l TPC-C
l TPC-D
l TPC-H
l TPC-E Benchmark (page 284)
The Benchmark properties page allows you to change scale factors properties.
To Change Scale Factors
You can set the load test scale factor when:
l Creating a new script
l Creating/delete a benchmark object, using the Benchmark Object Wizard
l In Benchmark Factory settings
l From the Benchmark Object's property page
-
Benchmark Factory for Databases User GuideHow Do I...
104
Setting the Scale Factor in Benchmark Factory Settings
Benchmark Factory settings allows you to set default scaling factors so you don't have to set iteach time you create a new load test.
To set a default scale factor
1. Select Edit | Settings. The settings dialog displays.
2. Click Benchmarks.
3. Double-click the desired benchmark. The Benchmark Properties dialog displays.
4. Set the desired scale in Benchmark Scale.
5. Click OK.
6. Click OK.
Changing the Scale Factory from the Benchmark Object Property Page
After creating the a benchmark object, you can edit that object to change the scale factor.
To edit the scale factor for a Benchmark Object
1. Click on the desired Benchmark Object in the Script view.
2. In the properties view for the Benchmark Object, select Edit. The Benchmark Propertiespage displays.
3. Set the desired scale in Benchmark Scale.
4. Click OK.
-
Benchmark Factory for Databases User GuideHow Do I...
105
Execute File WizardDuring the running of a workload, you may want to add an executable file (.bat or .exe) as partof a job. This gives you the flexibility of setting up your database before running a workload.For example, you could run a .bat file to set up a prepare SQL statement before running aworkload.
To execute a file:
1. Select the Job or Load Scenario node.
2. Select Wizards | Execute File.
3. Click Next. The Execute File dialog displays.
4. Enter the path or navigate to the desired file.
5. Click the Enforce Timeout checkbox if desired. Enforce Timeout allows you to enter thedesired number of seconds Benchmark Factory waits for the execution to complete beforecontinuing the job.
6. Enter the desired number of seconds.
7. Click Next. The completion dialog displays.
8. Click Finish. The Execute File is created and displays in the Script View.
BFScript WizardNote: Controls with yellow background can be scripted.
Benchmark Factory scripting all