TPC Benchmarks

Sandeep GonsalvesCSE 8330 – Project 1

SMU

May 1, 2004

Overview

Benchmarks Benchmark Wars Establishment of the TPC TPC Benchmarks: A, B, C, D, E, H, R,

S, W Conclusion

Benchmarks

Standard for comparison of various systems performing similar operations

Set of programs that simulate a typical workload on a given system

Used to measure system performance Metrics such as speed, performance,

price etc. are recorded Used to determine the optimal system

Domain-specific benchmarks

“No single metric can measure the performance of computer systems on all applications” [1]

Performance of a system varies tremendously from one application domain to another [1]

Domain-specific benchmarks

Key criteria for a domain-specific benchmark to be useful: Relevant Portable Scaleable Simple [1]

Early Benchmarks

TP1 benchmark Wisconsin benchmark

Benchmark Wars

Benchmark wars happen when one vendor publishes superior results for an important benchmark evaluation and the other vendors or individuals try to get back by improving their numbers [2]

Consequences: Abnormal results Waste of resources Confusion !

Initial Efforts

Working Group on Performance Measurement Standards (WG-PMS) [2]

Anon et.al. 1985 Three standard performance tests

for OLTP systems Most popular – DebitCredit [2]

Establishment of the TPC

Motivation: Need for the competition to get

civilized Put an end to manipulations Stop the confusion in the industry Form an industry forum to enforce

OLTP performance measurement standards [2]

TPC

TPC is the acronym for the Transaction Processing Performance Council

Established on 10th August, 1988 Founder: Omri Serlin Co-founder: Tom Sawyer Initial 8 members: Control Data Corp,

Digital Equipment Corp, ICL, Pyramid Technology, Stratus Computer, Sybase, Tandem Computers and Wang Laboratories [2]

TPC

First standard TPC-A published in November, 1989

TPC-A – Council’s version of the DebitCredit benchmark test [2]

Published the second standard TPC-B in August 1990

TPC-B – Council’s version of the TP1 benchmark test [2]

TPC

Major changes brought about by the TPC: [2]

Submission of a Full Disclosure Report (FDR)

Results must be audited by a TPC certified individual

TPC Members - 2004

TPC Auditors - 2004

TPC Benchmark ™ A Issued in November 1989 Obsolete as of 6th June, 1995 Measure performance of update-

intensive database environments [5]

e.g.. OLTP applications with characteristics like: “Multiple on-line terminal sessions Significant disk input/output Moderate system and application execution time Transaction integrity” [5]

TPC Benchmark ™ A Measures the transactions per second (tps) of

a system as a measurement of performance [5]

Measure the performance of systems in a wide area or local area network configuration [5]

Metrics are: “TPC-A local Throughput” and “TPC-A wide Throughput” measured in tps [6]

System configurations, details of the benchmark run, total cost mandatory part of FDR [6]

TPC-A Specifications [5]

Specification consists of 11 clauses TPC-A is stated in terms of a hypothetical

bank that has one or more branches and each branch has multiple tellers

Each customer of the bank has an account Transactions denote customer operations

such as withdrawals, deposits etc. performed by a teller at a branch

TPC-A Specifications

Fig 4: ER Diagram of the components of the TPC Benchmark™ A database [5]

TPC-A Specifications [5]

Any commercially available database management system (DBMS), database server, file system, etc can be used to implement the database

The system under test must support the ACID properties of transaction processing systems

Horizontal partitioning of files/tables is permitted

Vertical partitioning of files/tables is not permitted

TPC-A Results [5]

Throughput of the system in units of transactions per second is: “tpsA-Local” - local area networks “tpsA-Wide” - wide area networks

Cost is given as price/tpsA “ what is tested and/or emulated is priced

and what is priced is tested and/or emulated ”

5 year maintenance pricing must be included

TPC-A FDR Requirements [5]

Identification of sponsor of the benchmark Participating companies Program listings List of settings for parameters and

options that are tunable by a customer and have been altered from the defaults in actual products

Auditor’s name, address, phone number along with a copy of the auditor’s attestation letter

Etc…

TPC-A FDR Requirements [5]

In short:All that a customer would need to replicate the results !

The FDR document must be available to the public at a reasonable cost

Official language is English

TPC Benchmark ™ B

Approved in August 1990 Obsolete as of 6th June, 1995 Quite different from TPC-A Database Stress Test Not OLTP oriented It focuses on database management

systems (DBMS) applications and on the back-end database server [8]

TPC-B Specifications [8]

Specified in terms of a hypothetical bank Measures the total number of

simultaneous transactions that a system can carry out

No users, communication lines or terminals used

Equivalent to electronic data processing (EDP) batch processing applications

Metrics –Throughput: “tpsB” and the associated price-per-TPS

TPC Benchmark ™ C

Approved on July 23, 1992 Currently in use with its latest version 5.2 It is an OLTP benchmark It is centered on the transactions of a

wholesale supplier managing orders which is an order-entry environment [14]

TPC-C metrics are new-order txn rate (tpmC) and price/performance ($/tpmC) [14]

TPC Benchmark ™ C

It measures the entire business operation and is a more extensive and complex yardstick for measuring the performance of an OLTP system [14]

Current version is the 20th revision since 1992 [12]

TPC Benchmark ™ D

Approved on April 5, 1995 Obsolete as of April 6, 1999 Decision support benchmark Metrics are:

TPC-D Composite Query-per-Hour Metric (QphD@Size)

TPC-D Price/Performance ($/QphD@Size)

Systems Availability Date [18]

TPC Benchmark ™ D

“It illustrates decision support systems that: Examine large volumes of data; Execute queries with a high degree of

complexity; Give answers to critical, frequently-asked

business questions [18] ”

It’s successors are TPC-H and TPC-R

TPC Benchmark ™ H

Approved on February 26, 1999 Current version is 2.1.0 which was

specified on August 14, 2003 Decision support benchmark It executes a set of queries against a

standard database under controlled conditions to evaluate the performance of various decision support systems [20]

TPC Benchmark ™ H

Metrics are: TPC-H Composite Query-per-Hour Metric

(QphH@Size) which is the performance metric

TPC-H Price/Performance ($/QphH) which is the price-performance metric

The Systems Availability Date [20]

TPC Benchmark ™ R

Approved on February 26, 1999 Current version is 2.1.0 which was

specified on August 14, 2003 Decision support benchmark Similar to TPC-H

TPC Benchmark ™ R

The TPC-R metrics are: TPC-R Composite Query-per-Hour

Metric (QphR@Size) which is the performance metric

TPC-R Price/Performance ($/QphR) which is the price-performance metric

System Availability date [23]

TPC Benchmark ™ W

TPC’s latest benchmark First version 1.0 was approved on

December 9, 1999 Transactional web e-Commerce

benchmark that models a retail store on the internet like an online bookstore. [24]

TPC-W enables the testing of environments where a host of servers perform different functions [25]

TPC Benchmark ™ W TPC-W metrics are:

Number of web interactions processed per second (WIPS) which is the performance metric.

The associated price per WIPS ($/WIPS) The availability date of the priced

configuration [26]

It can verify the performance measurements of a variety of e-commerce servers in a real world internet environment [25]

Aborted benchmark efforts

TPC – S Server version of the TPC-C Did not receive sufficient Council

support Potential for abnormally high

performance ratings [22]

Aborted benchmark efforts TPC – E

“Enterprise” benchmark No complex benchmark available that

could stress large enterprise systems Could cause confusion Would cause vendors to spend

additional resources Was germane to only a very small

number of vendors competing in that arena [22]

What's up with the TPC

As of March 2004: Working on a new TPC-E

specification Working on a new decision support

benchmark: TPC-DS Carrying out revisions on the TPC-

H, TPC-R, TPC-W

Conclusion - TPC

Fair-competition in the industry End to bench marketing wars Widely used by all vendors Continuous evolution of its

benchmarks

Conclusion - TPC Log

References [1] Jim Gray: Database and Transaction Processing

Performance Handbook. The Benchmark Handbook 1993. Online edition. http://www.benchmarkresources.com/handbook/index.html

[2] Omri Serlin: The History of DebitCredit and the TPC. The Benchmark Handbook 1993. Online edition.

http://www.benchmarkresources.com/handbook/index.html

[3] Anon, et al, “A Measure of Transaction Processing Power”, Datamation, V. 31.7, April 1985, pp. 112-118.

[4] Gray, J.N., Reuter, A., “Transaction Processing: Concepts and Techniques, Morgan Kaufmann, San Mateo, CA, 1993, pp. 11-12, 168.

References [5] TPC BENCHMARK™ A Standard

Specification Revision 2.0 7June 1994 http://www.tpc.org/tpca/spec/tpca_current.pdf

[6] TPC-Ahttp://www.tpc.org/tpca/default.asp

[7] Hanson, Robert J., TPC Benchmark B - What It Means and How to Use It, AT&T Global Information Solutionshttp://www.tpc.org/tpcb/default.asp

[8] TPC BENCHMARK™ B Standard Specification Revision 2.0 7June 1994 http://www.tpc.org/tpcb/spec/tpcb_current.pdf

References [9] Bramer, Brian. System benchmarks,

DeMontfort University, UK. http://www.cse.dmu.ac.uk/~bb/Teaching/ComputerSystems/SystemBenchmarks/BenchMarks.html#introduction

[10] Levine, Charles.,SIGMOD '97 Industrial Session 5 - 5/29/97. http://www.tpc.org/information/sessions/sigmod/indexc.htm

[11] Ozsu, T., P. Valderez’s, Principles of Distributed Database Systems, Second Edition. Prentice Hall, Englewood Cliffs, NJ., 1999.

[12] TPC BENCHMARK™ C Standard Specification Revision 5.2 December 2003 http://www.tpc.org/tpcc/spec/tpcc_current.pdf

References

[13] Patterson, D. A., J. L. Hennessy, Computer Architecture, a Quantitative Approach, Morgan Kaufmann, San Mateo, CA, 1990. Chapter 1.

[14] Raab, Francois et.al. Overview of the TPC Benchmark C: The Order-Entry Benchmark.http://www.tpc.org/tpcc/detail.asp

[15] TPC-C http://www.tpc.org/tpcc/default.asp

[16] TPC-Dhttp://www.tpc.org/tpcd/default.asp

References [17] Stephens, Jack. TPC-D. The Industry

Standard Decision Support Benchmarkhttp://www.tpc.org/information/sessions/sigmod/indexc.htm

[18] TPC BENCHMARKTM D (Decision Support) Standard Specification Revision 2.1http://www.tpc.org/tpcd/spec/tpcd_current.pdf

[19] TPC-Dhttp://www.tpc.org/tpcd/default.asp

[20] TPC BENCHMARK™ H (Decision Support) Standard Specification Revision 2.1.0http://www.tpc.org/tpch/spec/tpch2.1.0.pdf

References [21] Levine, Charles. TPC Benchmarks, Microsoft.

http://research.microsoft.com/~gray/WICS_96_TP/

[22] Draft White Paper. August 2, 1999. Object Management Group www.omg.org/docs/bench/99-08-02.ps

[23] TPC-Rhttp://www.tpc.org/tpcr/default.asp

[24] TPC Benchmark™ W (Web Commerce) SpecificationVersion 1.8. Feb 19, 2002.http://www.tpc.org/tpcw/spec/tpcw_V1.8.pdf

References

[25] Smith, Wayne D. TPC-W*: Benchmarking: An Ecommerce Solution. Revision 1.2. Intel Corporation.http://www.tpc.org/tpcw/TPC-W_wh.pdf