© Copyright 2008, SoftWell Performance AB
1
Performance Testing Distributed Systems
Concepts and Terminologyv0.6.1
© Copyright 2008, SoftWell Performance AB
2
Performance Testing - Concepts and Terminology
The document structure
The document describes performance testing in two parts:
General performance concepts
Performance testing concepts
© Copyright 2008, SoftWell Performance AB
3
Performance Testing - Concepts and Terminology
The document structure
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
Generalperformance
concepts 7. Abstract Performance metrics
6. Measurement data objectives and attributes
5. Measured objects
4. Performance characteristics
8. Performance data processing
9. General performance test concepts
10. Performance test environment
11. Performance test specification concepts
12. Workload concepts
Performancetesting
concepts
© Copyright 2008, SoftWell Performance AB
4
Part 1General performance concepts
© Copyright 2008, SoftWell Performance AB
5
Performance Testing - Concepts and Terminology
Performance characteristics
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
4. Performance characteristics
Categorizing performance characteristics
Powerfulness characteristics
Reliability characteristics
Efficiency characteristics
© Copyright 2008, SoftWell Performance AB
6
Performance Testing - Concepts and Terminology
Measured objects
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
5. Measured objects
Measured services
Measured components
Service concepts
Service characteristics
Service interfaces
© Copyright 2008, SoftWell Performance AB
7
Performance Testing - Concepts and Terminology
Measurement data objectives and attributes
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
6. Measurement data objectives and attributes
Performance metric objectives
Measurement data attribute sets
Identification attributes
Unit attributes
Conditional attributes
Processing attributes
Metric types
© Copyright 2008, SoftWell Performance AB
8
Performance Testing - Concepts and Terminology
Abstract performance metrics
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
7. Abstract Performance metrics
Abstract powerfulness metrics
Abstract reliability metrics
Abstract efficiency metrics
© Copyright 2008, SoftWell Performance AB
9
Performance Testing - Concepts and Terminology
Performance data processing
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
8. Performance data processing
Steps in performance data processing
Collection and storage of raw performance data
Condensation and normalization of raw performance data
Performance data computations
Evaluation of performance data
Presentation of performance data
© Copyright 2008, SoftWell Performance AB
10
Part 2 Performance testing concepts
© Copyright 2008, SoftWell Performance AB
11
Performance Testing - Concepts and Terminology
General performance test concepts
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
9. General performance test concepts
Performance test concepts
Performance test phases
Performance test objectives
Performance objectives and performance requirements
Performance measurement conditions
Performance targets
Performance measurement standards
Some performance measurement characteristics
© Copyright 2008, SoftWell Performance AB
12
Performance Testing - Concepts and Terminology
Performance test environment
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
10. Performance test environment
Test environment concepts
System Under Test concepts
Test System concepts
© Copyright 2008, SoftWell Performance AB
13
Performance Testing - Concepts and Terminology
Performance test specifications
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
11. Performance test specifications
Elements of performance test specifications
Test objectives
Test conditions
Test configurations
Test data specifications
Test evaluation specifications
© Copyright 2008, SoftWell Performance AB
14
Performance Testing - Concepts and Terminology
Workload concepts
MTS – Performance Testing Distributed Systems1 – Concepts and Terminology
12. Workload concepts
Workload set or Traffic set
Workload content
Workload volume
Load concepts
Workload time distribution
© Copyright 2008, SoftWell Performance AB
15
What is next?
© Copyright 2008, SoftWell Performance AB
16
Performance Testing – Next step
Proposal
The next document should describe
• Methods in all phases of performance tests
• How methods are applied
The purpose is to
• Describe methodology of performance testing in a generalized manner
• Describe how methods are applied •
• Start the groundwork on a framework for performance testing based on a common view of abstract performance metrics and characteristics
MTS – Performance Testing Distributed Systems2 – Methodology and Realization
© Copyright 2008, SoftWell Performance AB
17
MTS objectivesfor
Performance Testing
© Copyright 2008, SoftWell Performance AB
18
MTS objectives for Performance Testing
The MTS objectives should be to:
• Develop a formalized view of performance • Develop a formalized view of performance test methods • Develop a formalized view of performance tests
That can be used as:
• A framework for applied performance tests and test suites• A guideline for design of performance tests tools• A guideline for performance test standards
© Copyright 2008, SoftWell Performance AB
19
Some long term goalson
Performance andPerformance Testing
© Copyright 2008, SoftWell Performance AB
20
Long term goals for Performance Testing
Performance is an increasingly important aspect of a design or a system.Performance will be an increasingly important aspect of a standards. A validated function works under most circumstances in production, validated performance is not a bulletproof guarantee for production.
A formalized view of performance and performance testing enables:
• A complete specification of a system • Modeling and design of a system’s performance• Simple monitoring of performance in design and production• Far better ways for predicting performance in production• Intelligent performance test tools that can be built into the system• A continuously improved understanding of performance• A safer investment in software development
Some examples
© Copyright 2008, SoftWell Performance AB
21
“To measure is to know”
- Lord Kelvin