performance continuous integration
DESCRIPTION
TRANSCRIPT
![Page 1: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/1.jpg)
Telefónica Digital
TELEFÓNICA DIGITAL
Barcelona, March 21st 2013
![Page 2: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/2.jpg)
Telefónica DigitalProduct Development & Innovation
About: me
last 8 years i have been working as performance test engineer with different tools and enviroments.
2
![Page 3: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/3.jpg)
Architecture DesignWeb Performance Optimization
![Page 4: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/4.jpg)
The Past
No instruments
Users Review
Late or none Performance Testing
No Real User Monitoring
Reactive Performance Tuning
![Page 5: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/5.jpg)
No tools, no performance dashboard, performance is for sysadmins and operators
![Page 6: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/6.jpg)
Releases are costly and it may take several months of work
![Page 7: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/7.jpg)
Manual testing of each release after code freeze
![Page 8: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/8.jpg)
Non functional Requirements are most likely ignored
![Page 9: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/9.jpg)
In production there is no monitoring of the traffic and how it affects the business
![Page 10: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/10.jpg)
Users feedback is usually negative and there is no interaction with developers and designers
![Page 11: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/11.jpg)
Application’s performance affects directly to market’s performance
![Page 12: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/12.jpg)
• Continuous Integration– Functional testing– Automation– Monitoring
The Present
![Page 13: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/13.jpg)
Continuous Integration for functional testing is working already in nightly builds
![Page 14: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/14.jpg)
Automation reduces time to market for the applications
![Page 15: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/15.jpg)
Monitoring the real user behaviour and not just healthcheck of servers
![Page 16: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/16.jpg)
Error and risks management
![Page 17: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/17.jpg)
Tuning and bugfixing
![Page 18: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/18.jpg)
Listening to user feedback
![Page 19: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/19.jpg)
The Future
• Continuous Performance Integration– Performance tests integrated in Jenkins– Automation of the trend reports– Real User Monitoring Real time feedback
![Page 20: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/20.jpg)
Telefónica DigitalProduct Development & Innovation
SCRUM and PERFORMANCE
20
![Page 21: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/21.jpg)
Pruebas de RendimientoProactive Performance testing for each release. Load tests will discover the flaws and bottlenecks, the application or the systemmay have in production environment
TestingTesting
![Page 22: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/22.jpg)
Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.High availability of the application and the system is the goal of a ready for service status. The application and the systems must be stable , efficient and dimensioned according to the usage.
AvailabilityAvailability
![Page 23: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/23.jpg)
Not only the response time is important, an intelligent use of the resources is needed to grow in the future. Efficiency, understood as capacity to dispose of the system resources to achive an objective, in our case response time and uptime
VelocityVelocity
![Page 24: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/24.jpg)
Being able to grow depending on the necessities of the market, users and new technologies is one of the focus for a performance engineer
ScalabilityScalability
![Page 25: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/25.jpg)
A performance test is easy. It is easy to design non realistic scenarios. It is easy to collect irrelevant data. Even with a good scenario and Appropiate data, it is easy to use and incorrect statistic method to analysis the results. - Alberto Savoia
ScenariosScenarios
![Page 26: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/26.jpg)
One of the most important parts of a good performance test design is to have an appropiate load test environment, as similar as possible to Production at all levels, networking, systems and application architecture.
PreProductionPreProduction
![Page 27: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/27.jpg)
Otro título
EscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenarios
To know the production environment is key to take good decisions about how to design a performance test plan. Designing a plan according To real traffic and usage of the platform is key in creating validation criteria
MonitoringMonitoring
![Page 28: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/28.jpg)
Developers, DBA's, QA's, DevOps, product owners ... All the team is part of performance
Performance TeamsPerformance Teams
![Page 29: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/29.jpg)
Otro título
There are many tools available in the market for load testing and monitoring. An effort in evaluating these tools will benefit at long term the Execution of the tests. However, the most important part is how the reports are generated and who is going to interpret them.
ToolsTools
![Page 30: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/30.jpg)
Otro título
Mas puntos
Not only unique users or session times are important. How the users work with the application and the psicology of the them are key to Understand the results and how it affects to business.
Real User MonitoringReal User Monitoring
![Page 31: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/31.jpg)
Keep it simple, use cache wisely, invest in testing and monitoring, create a culture of performance in all the organization
Best PracticesBest Practices
![Page 32: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/32.jpg)
Tuning
Techonology develops at high speed. To bring out the best of our product, business and techonology need to evolve by the hand. Investing in Performance research is crucial to keep up with other internet competitors.
InnovationInnovation
![Page 33: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/33.jpg)
![Page 34: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/34.jpg)
Telefónica DigitalProduct Development & Innovation
Understand the Project Vision and ContextProject Vision
Project Context
Understand the system
Understand the Project Environment
Understand the Performance Build Schedule
34
![Page 35: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/35.jpg)
Telefónica DigitalProduct Development & Innovation
Identify Reasons for Testing Performance
Success Criteria• Application performance requirements and goals
• Performance-related targets and thresholds
• Exit criteria (how to know when you are done)
• Key areas of investigation
• Key data to be collected
35
![Page 36: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/36.jpg)
Telefónica DigitalProduct Development & Innovation
Identify the Value Performance Testing Adds to the ProjectIn general, the types of information that may be valuable to
discuss with the team when preparing a performance-testing strategy for a performance build include:
› The reason for performance testing this delivery
› Prerequisites for strategy execution
› Tools and scripts required
› External resources required
› Risks to accomplishing the strategy
› Data of special interest
› Areas of concern
› Pass/fail criteria
› Completion criteria
› Planned variants on tests
› Load range
› Tasks to accomplish the strategy
36
![Page 37: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/37.jpg)
Telefónica DigitalProduct Development & Innovation
Configure the Test Environment
Set up isolated networking environment
Procure hardware as similar as possible to production
Coordinate bank of IP’s for IP spoofing
Monitoring tools and operating systems like production
Load generation tools or develop your own
37
![Page 38: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/38.jpg)
Telefónica DigitalProduct Development & Innovation
Identify and Coordinate Tasks
Work item execution method
Specifically what data will be collected
Specifically how that data will be collected
Who will assist, how, and when
Sequence of work items by priority
38
![Page 39: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/39.jpg)
Telefónica DigitalProduct Development & Innovation
Execute Task(s)
Keys to Conducting a Performance-Testing Task
• Analyze results immediately and revise the plan accordingly.
• Work closely with the team or sub-team that is most relevant to the task.
• Communicate frequently and openly across the team.
• Record results and significant findings.
• Record other data needed to repeat the test later.
• Revisit performance-testing priorities after no more than two days.
39
![Page 40: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/40.jpg)
Telefónica DigitalProduct Development & Innovation
Analyze Results and Report
pause periodically to consolidate results
conduct trend analysis
create stakeholder reports,
pair with developers, architects, and administrators to analyze results
40
![Page 41: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/41.jpg)
But … what are you actually doing day by day?
![Page 42: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/42.jpg)
Telefónica DigitalProduct Development & Innovation
Case of Study
42
![Page 43: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/43.jpg)
![Page 44: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/44.jpg)
Selenium Framework
![Page 45: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/45.jpg)
Selenium Framework
// Set default NetExport preferences profile.setPreference(domain + "netexport.alwaysEnableAutoExport", true);profile.setPreference(domain + "netexport.showPreview", false);profile.setPreference(domain + "netexport.beaconServerURL", "http://localhost/har-db");profile.setPreference(domain + "netexport.autoExportToFile", false);profile.setPreference(domain + "netexport.autoExportToServer", true);profile.setPreference(domain + "netexport.sendToConfirmation", false); //set net export preferences \\showslow directory to save .har files//profile.setPreference(domain + "netexport.autoExportToFile", true);//profile.setPreference(domain + "netexport.defaultLogDir", "C:\\showslow\har\\");
![Page 46: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/46.jpg)
V1.0 Branch BBranch A
Xms
Xms + 100ms Xms - 50ms
Branches comparison
![Page 47: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/47.jpg)
Performance trends
![Page 48: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/48.jpg)
![Page 49: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/49.jpg)
![Page 50: Performance Continuous Integration](https://reader033.vdocuments.us/reader033/viewer/2022061212/549535f2b47959654d8b4cbf/html5/thumbnails/50.jpg)