![Page 1: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/1.jpg)
Performance Engineering in Agile and Devops
Gopal BrugalettePerformance ArchitectSoFi
![Page 2: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/2.jpg)
2
$30 billion in funded loans
700,000 members and counting98% would recommend SoFi to a friend
![Page 3: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/3.jpg)
Gopal BrugalettePerformance Architect
![Page 4: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/4.jpg)
Performance Engineering in Waterfall
Design Build Test Perf Test Deploy
Design Build Test Deploy Perf Test
OR
![Page 5: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/5.jpg)
Performance testing challenges
Environment
• Build/Maintain
• Deployment
• Dependencies
• External systems
• Matching Production
• Support
Data
• Complex Models
• Volumes
• New Data
• PII/PCI
Scripting
• What changed?
• Old scripts break
• New scripts needed
• Coverage %
Time Reliability
![Page 6: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/6.jpg)
Perf as Non-Functional Requirement
Does not work
Nobody works on it
(Ignoring) doesn’t work for Customer
![Page 7: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/7.jpg)
Performance as a Feature
Conversions Rates vs Load Times
![Page 8: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/8.jpg)
Responsibility for Performance
QA Dev(ops)Performance
Engineers
(Dev)OpsProduct
ManagementMarketing/Sales
![Page 9: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/9.jpg)
Performance early, often, everywhere
Business Drivers
Design Build Test Prod
![Page 10: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/10.jpg)
Business & Marketing
Growth projections
New product launches
Marketing Campaigns
Advertising
Tie-ins and Cross Promotions
Customer Experience
![Page 11: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/11.jpg)
Planning and Requirements
SLA’s, SLO’s
User Flows, Customer Journey
Workloads
Observability
Scripting Needs
![Page 12: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/12.jpg)
Performance in Agile Process
Create Performance Stories & Tasks
Add a Definition of Done
Evaluate each story for performance
impacts
Create/update scripts
Update the workload model
Identify/create/load data
Identify metricsCreate Monitoring dashboards/alerts
![Page 13: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/13.jpg)
Design
Scalability
Performance
Testability
Observability
Resiliency
![Page 14: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/14.jpg)
Build
Observability
• Enumerate
• Log useful info
• Don’t log useless info
• Traceability
Resiliency
• Timeouts
• Retries
• Handle failures
![Page 15: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/15.jpg)
CI/CD
Shift left, test Early
Quick Tests
Baselines response times
Counts (Service calls, DB calls)
Catch big issues here
Catch small issues later
![Page 16: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/16.jpg)
Build a framework
Make it easy to script, execute, monitor and analyze
![Page 17: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/17.jpg)
Lab Load/Stress Testing
GrowthEndurance &
StabilityFull Analysis
Major Architectural
Changes
Unique Peak Events
![Page 18: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/18.jpg)
Test in Production
A/B
• Run multiple versions
• Compare performance & behavior
• Informed decisions
Canary
• Soft Launch
• Ramp Up
• Monitor
Synthetic Load
• Full Scale Testing
• Better you than them
• Enable testability
RUM
• Browser-based
• Real data
• All combinations & flows
![Page 19: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/19.jpg)
Monitoring
Customer Experience
Business Relevance
Metrics
Dashboards
Alerts
Use ML
![Page 20: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/20.jpg)
Agile Performance Engineering
Individuals and interactions over process and tools
![Page 21: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/21.jpg)
When Scrum?
Strongly Aligned with dev
Predictable, planned work
Team that needs structure
Inexperience with managing work, agile
Company in transition/transformation stage
Keep sprints short
Consider value of activities
![Page 22: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/22.jpg)
Kanban for Performance
Unpredictable workload
Fluctuating priorities
Similar work across
engagements
Use two levels (Epic/Story)
Make stories small
Keep workflow
simple
Manage backlog
Manage WIP
![Page 23: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/23.jpg)
Swarming & Mobbing
Combine and conquer
Share expertise and knowledge
A working session – not status or a standup
Focus on one task
Expanded skillsets and backgrounds
Builds team
Engage everyone
Manage your WIP
![Page 24: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/24.jpg)
Build a DevOps Team
Architect /TPM
Technical leadership
Work management
Enterprise engagement
Vision
Perf Engineer
Analysis
Approaches
Script & Test
Tooling
Manager
HR
Escalation
Relationships
DevOps
Manage environment
Tooling
Analysis
Developer
Get bored
Lack Context
Develop complex solutions
QA
Different Context
Different Approach
Different Skillset
![Page 25: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/25.jpg)
Prioritized Responsibilities
FeaturesFunctional
CorrectnessRelease Dates
Architecture
Security Privacy Outages Performance
Availability Resiliency
![Page 26: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/26.jpg)
Win hearts and minds
Educate Enable Influence
![Page 27: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/27.jpg)
Keep Score
• Share Interesting Metrics
• Track the P1’s, P2’s and P3’s.
• Focus on Process Not People
Category Issue Count %
Configuration 2 3%
Deployment 4 5%
Feature Code 1 1%
Maintenance 1 1%
Operational 66 89%
Total 74
![Page 28: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/28.jpg)
Target the right teams
Business Critical Functionality
Large Volume transactions
Architecturally Critical Components
High risk technical changes
New Technology
![Page 29: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/29.jpg)
Get the party started
Kickstart
•Write Scripts
•Run Tests
•Analyze reports
Train
•Testing Tools
•APM tools
•Analysis
Handoff
![Page 30: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/30.jpg)
Whitepapers and Case Studies
Tell a story
Make it relevant
Technical Solutions
Business Outcomes
Customer Impact
Successes Failures
![Page 31: Performance Engineering in Agile and Devops - TestCon · Agile and Devops Gopal Brugalette Performance Architect SoFi 2 $30 billion in funded loans 700,000 members and counting 98%](https://reader031.vdocuments.us/reader031/viewer/2022013023/604ce60dae5eb075f23662cf/html5/thumbnails/31.jpg)
Reach me @[email protected]
Or LinkedIn31
Performance is a Feature
Business Drivers
Design Build Test Prod