agile software performance engineering · 1. publication of the aspe approach in the domain of...

Agile Software Performance Engineering

advised by Privatdoz. Mag. Dr. Manuel Wimmer (BIG)

www.johannes-artner.at/#ASPE

Johannes Artner, TU Wien - January 2017 Slide 1 of 26

Master Thesis, Johannes Artner:

Motivation & Problem

• Performance a key characteristic

• With regard to timeliness

• Useful if taken into account early [1]

• In practice: More art than science [2]

Johannes Artner, TU Wien - January 2017 2 of 26

Software Performance Engineering [3]

• Model-based approach

• Measurement-based approach


Aim of the work 1/2

Research interest:

Combination of model-based approach and measurement-based approach in SPE.

How can the model-based- and measurement-based approach in SPE be combined?


Vision / Tools

DevOps

Johannes Artner, TU Wien - January 2017

Agile development

ContinuousDelivery

Cloud Computing

Markovmodels

Queuing models

5 of 26

Web 2.0

Aim of the work 2/2• Within an agile software engineering workflow

• How can SPE-activities be integrated into an agile software engineering methodology?

• Metamodels for integrating predictive data and measurement data• What information is needed to evaluate a modern software systems performance?

• Evaluating the Markov assumption• With which techniques can the Markov assumption for user behaviour on software systems be evaluated?

• Approach to get from Markov models to queuing networks• How can queuing models be derived from user behaviour models in Markov-notation?

• Utility: Benefits, Tools & Overhead• What is the benefit of my approach and how much overhead is it? What tools and techniques are necessary to be

practically usable?


Methodology

• Systematic literature review• Kitchenham et al. [4]

• Proposition of a workfow model and two metamodels• Design Science: Hevner et al. [5]

• Evaluation in a Case Study• Höst and Runeson [6]


Related Work

• Intermediate formats: e.g.: CSM [7]

• Performance annotations: e.g.: MARTE [8]

• Transformation frameworks: e.g.: PUMA [9]

• Load test tools: e.g. JMeter [11]


Main problem: Approaches are adding overheadand/or complexity!

8 of 26

The ASPE approach (1/4) – CETO (Components Emission and Timely Observations)

• Resources

• Workloads

• Workload-intensity

• Static data

• Predictive data

• Measured data


The ASPE approach (2/4) – MUPOM (Markov Usage Process and Operation Measurements)


The ASPE approach(3/4) - Workflow

• Force automation

• Reuse artefacts

• Approximations instead of

over-engineering SPE


The ASPE approach (4/4) - Transformations

• Software models and observations to CETO

• CETO to MUPOM

• CETO to some performance model

• MUPOM to Queuing networks


From Markov models to Queuing networks

For ergodic systems where the Markov-assumption holds:

1. Little‘s Law to determine the number of users in the entire system

2. Steady state analysis to determine users in each state

3. Transition-rates (Lambda λ) between states


Case study: The Travelistr software system


Tool-Support

• OperationsTraceMonitor (self-developed)• Simple Java language level tool

• Operation-durations as well as user-traces to CSV-files

• UserTrace2Markov (self-developed)• Takes CSV-input from OperationsTraceMonitor

• Calculates first- as well as second order Markov transition rates

• Markov4JMeter

• R-project

• JMVA (JMT)


Case study Travelistr – Sprint 0 (1/4)


P Start Login Regis. Abo. Dash. Logo. Publ. Profi. Near. Like

Start 0.2 0.4 0.3 0.1 0 0 0 0 0 0

Login 0 0.2 0.1 0 0.7 0 0 0 0 0

Regis. 0 0.1 0.2 0.7 0 0 0 0 0 0

Abo. 0.6 0 0 0.4 0 0 0 0 0 0

Dash. 0 0 0 0 0.2 0.1 0.1 0.1 0.5 0

Logo. 0.8 0 0 0 0 0.2 0 0 0 0

Publ. 0 0 0 0 0.7 0 0.3 0 0 0

Profi. 0 0 0 0 0.2 0 0 0.1 0.3 0.4

Near. 0 0 0 0 0.2 0 0 0.1 0.3 0.4

Like 0 0 0 0 0 0.2 0 0 0.7 0.1

16 of 26



P Steady state E(N) λ per sec 1 / λ

Start 0.106 3.18 2.562 0.39

Login 0.059 1.77 1.413 0.71

Register 0.047 1.41 1.131 0.88

About 0.073 2.19 0.318 3.14

Dashboard 0.145 4.35 3.486 0.29

Logout 0.052 1.56 0.435 2.30

Publish 0.021 0.63 0.435 0.75

Profile 0.064 1.92 1.388 0.14

Nearby 0.301 9.03 6.909 0.28

Like 0.134 4.02 3.612 0.28

17 of 26


Predicted arrival rates and operation durations (in ms):


Operation Lambda λ DB CPU ImageServer

handleRequest() 21.639 0 500 0

getUser() 3.882 750 500 0

saveUser() 1.131 900 500 0

publishImage() 0.435 600 2000 1000

getNearby() 6.909 1500 500 500

doLike() 3.612 700 500 0

18 of 26


Mean Value Analysis - Utilization


* Aggregate handleReq getUser() saveUser() publishImg getNearby doLike

DB 17.0823 0.0 2.9115 1.0179 0.261 10.3635 2.5284

CPU 19.4565 10.8195 1.941 0.5655 0.87 3.4545 1.806

ImageServ. 3.8895 0.0 0.0 0.0 0.435 3.4545 0.0

19 of 26

Case study Travelistr – Sprint 1

• Operations: handleRequest(), getUser() and saveUser()

• Load tests (Markov4JMeter)

• Enrich model with measured data (DB: 13.28, CPU: 6.28, ImageS.: 3.89)



• Operation: • uploadImage()

• scaleImage()

• uploadToStore()

• saveImageRef()

• Load tests (Markov4JMeter)

• Enrich model with measured data

(DB: 13.05, CPU: 6.06, ImageS.: 3.84)



• Operations: getNearby() and doLike()

• Running load tests (Markov4JMeter)

• Enrich model with measured data

(DB: 2.33, CPU: 0.69, ImageS.: 0.39)


Empiric user test


Empiric user test – Markov assumption

Approximative approach based on work of Li et al. [12]


State Information loss Observed Transitions

Nearby 4.36% 1067

Dashboard 11.45% 506

Profile 6.72% 193

Publish 2.87% 190

Published 1.77% 153

Start 7.58% 128

Login 3.14% 106

Register 12.48% 35

25 of 26

Utility / Conclusion

• ASPE approach showed high utility• Effects of changes or decisions visible already during development

• Very simple way of describing user behaviour with Markov models

• Difficult to predict operation duration.

• High demand for tool support and automation!


Appendix

Johannes Artner, TU Wien - January 2017 27

Future Work (1/2)

• Can observed user-interactions on a limited set of features be extrapolated to a systems entire user behaviour model?

• How can orderings of operations as well as execution-probabilities be derived from observation data?

• How can results of the ASPE approach be incorporated to Elastic Cloud concepts?

• Given incomplete information, Hidden Markov Models seem to be also useful in

SPE and the ASPE approach.


Future Work (2/2)

1. Publication of the ASPE approach in the domain of web-engineering.

2. Bringing the utility of the ASPE approach to software engineeringpractitioners in form of a round-trip, web-based evaluation tool…• .. that integrates the ASPE approach as well as the userTrace2Markov-Tool

3. Extend the userTrace2Markov-tool with respect to full automation.


CETO to MUPOM Transformation1. (Optional) Classify users and group users of same types.

2. (Optional) Unveiling the underlying Markov chain.

3. Canonicalization of operation durations

4. Constructing probability distributions for think times and operationdurations.

5. Constructing the transition matrix of the Markov model.

6. Check whether the Markov-property for the usage holds.

7. Calculate the steady state distribution.

8. (Optional) Derive the causal orderings and parallelism of operations.

9. Check whether the system satisfies the product-form assumption ofqueuing networks.

10. Construct a performance model.Johannes Artner, TU Wien - January 2017 30

Canonicalizations of operations


ResourcesResources

15:00 15:15

15:06 15:12

15:06 - 15:12

Use-case 7

15:06 - 15:12

Use-case 7

15:15

End of observation

15:15

End of observation

15:03 - 15:06

Use-case 1

15:03 - 15:06

Use-case 1

Observed behaviour of a single user

Observed operations on two different resources for use-case 7

15:12 - 15:14

Use-case 2

15:12 - 15:14

Use-case 2

15:00 - 15:03

Use-case 4

15:00 - 15:03

Use-case 4

15:09 - 15:10

Operation 1

15:09 - 15:10

Operation 1

15:06 - 15:08

Operation 3

15:06 - 15:08

Operation 3

15:06 15:12

15:06 - 15:07

Operation 5

15:06 - 15:07

Operation 5

15:10 - 15:11

Operation 4

15:10 - 15:11

Operation 4

15:08 - 15:10

Operation 2

15:08 - 15:10

Operation 2

15:00

Start of observation

15:00

Start of observation

31

Hidden Markov Models


Threads to validity – Markov assumption test

• 32 users in empiric user test

• Only states with more than 20 transitions were taken into account

• Invitations only to „trusted“ people

• Gamified approach („most liked pictures of the day“)


Hypotheses (1/2)

• Hypothesis 1 (accepted): The agile methodology in combination withcontinuous delivery, model-driven engineering and the use of the Markovmodel formalism for describing workloads on resources is a valid approachto combine predictive- and measurement-based approaches in SPE.

• Hypothesis 2 (accepted): The transition probability between pages/featuresof a typical Web 2.0 application can be described in Markov modelnotation and in consequence fulfills the Markov assumption.

• Assumption 1 (accepted): Typical Web 2.0 applications are very likely to beexpressed in terms of ergodic systems. And if not, they can be transferredto such.


Hypothesis (2/2)

• Hypothesis 3 (accepted): Users in typical Web 2.0 applications are not bound to a specific starting state. Users are likely to enter at anystate.

• Hypothesis 4 (could not be verified): The fraction of users in each statein the long run, givent that the Markov assumption holds, will be closeto the steady state solution of the user behaviour expressed in Markovmodel notation.

• Hypothesis 5 (rejected): In order to retrieve service times ofoperations on resources, it is sufficient to measure response times on a language level.


Literature[1] L. G. Williams and C. U. Smith. Performance evaluation of software architectures. In Proceedings of the 1st international workshop on Software and performance, pages 164 -177. ACM, 1998.

[2] M. Harchol-Balter. Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, New York, NY, USA, 1st edition, 2013.

[3] M. Woodside, G. Franks, and D. C. Petriu. The Future of Software Performance Engineering. Future of Software Engineering (FOSE '07), pages 171 - 187, 2007.

[4] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil. Lessons from applying the systematic literature review process within the software engineering domain. Journal of systems and software, 80(4):571{583, 2007.

[5] R. H. Von Alan, S. T. March, J. Park, and S. Ram. Design science in information systems research. MIS quarterly, 28(1):75{105, 2004.

[6] P. Runeson and M. Höst. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering, 14(2):131{164, 2009.

[7] D. B. Petriu and M. Woodside, “An intermediate metamodel with scenarios and resources for generating performance models from UML designs,” in Software and Systems Modeling, vol. 6, pp. 163–184, 2007.

[8] “Uml marte specification.” http://www.omg.org/spec/MARTE/1.1/. Accessed: 2016-06-30.

[9] M. Woodside, D. C. Petriu, D. B. Petriu, H. Shen, T. Israr, and J. Merseguer, “Performance by unified model analysis (PUMA),” Proceedings of the 5th international workshop on Software and performance WOSP 05, vol. pp, pp. 1–12, 2005.

[10] A. Van Hoorn, M. Rohr, and W. Hasselbring, “Generating probabilistic and intensityvarying workload for web-based software systems,” in SPEC InternationalPerformance Evaluation Workshop, pp. 124–143, Springer, 2008.

[11] M. Rohr, A. Van Hoorn, J. Matevska, N. Sommer, L. Stoever, S. Giesecke, and W. Hasselbring, “Kieker: continuous monitoring and on demand visualization of java software behavior,” in Proceedings of the IASTED International Conference on Software Engineering, SE 2008, pp. 80–85, 2008.

[12] Z. Li and J. Tian, “Testing the suitability of markov chains as web usage models,” in Computer Software and Applications Conference, 2003. COMPSAC 2003. Proceedings. 27th Annual International, pp. 356–361, IEEE, 2003.


agile software performance engineering · 1. publication of the aspe approach in the domain of...

Documents