Business Value ofAgile Testing
Dr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFE, DEVOPS
Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf
Dave’s NEW Business Agility Video: https://www.youtube.com/watch?v=-wTXqN-OBzADave’s NEWER Development Operations Security Video: https://vimeo.com/214895416DoD Fighter Jets vs. Amazon Web Services: http://davidfrico.com/dod-agile-principles.pdf
Scaling Up to Billion User Global Systems of Systems Using TDD, CONTINUOUS INTE-
GRATION & DELIVERY, and DEVOPS
Author Background Gov’t contractor with 34+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe
2
Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 175+ talks to 13,800 people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at six Washington, DC-area universities
Today’s Whirlwind Environment
3
OverrunsAttritionEscalationRunawaysCancellation
GlobalCompetition
DemandingCustomers
OrganizationDownsizing
SystemComplexity
TechnologyChange
VagueRequirements
Work LifeImbalance
InefficiencyHigh O&MLower DoQVulnerableN-M Breach
ReducedIT Budgets
81 MonthCycle Times
RedundantData Centers
Lack ofInteroperability
PoorIT Security
OverburdeningLegacy Systems
ObsoleteTechnology & Skills
Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
Today’s Global Marketplace Most of world’s population connected to Internet Systems must support billions of simultaneous users New approaches are needed to scale to global market
4Kemp, S. (2016). Digital in 2016: We are social's compendium of global digital, social, and mobile data, trends, and statistics. New York, NY: We Are Social, Inc.
Software in U.S. DoD Systems
Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.
No. of software-intensive systems is growing 80% of US DoD functions performed in software Major driver of cost, schedule, & tech. performance
5
Software in U.S. DoD Avionics
Blackburn, M. R. (2014). Transforming systems engineering through a holistic approach to model centric engineering. Washington, DC: Stevens Institute of Technology.
Software in U.S. DoD avionics growing exponentially 10x growth from F-16 to F-22 (& another 10x to F-35) Productivity must grow by 10x for next gen systems
6
Requirements Defects & Waste
7Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all
Other 7%
Requirements47%
Design28%
Implementation18%
Defects
Always 7%
Often 13%
Sometimes16%
Rarely19%
Never45%
Waste
Traditional Projects
8
Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DE
FEC
TS
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
SIZE
Size vs. Productivity
PR
OD
UC
TIV
ITY
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
SIZE
Size vs. Change
CH
AN
GE
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
SIZE
Size vs. SuccessS
UC
CE
SS
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
SIZE
Global Project Failures
9Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trill
ions
(US
Dolla
rs)
Expenditures Failed Investments
0% 20% 40% 60% 80% 100%
28%
34%
29%
35%
32%
33%
27%
28%
29%
49%
51%
53%
46%
44%
41%
56%
55%
52%
23%
15%
18%
19%
24%
26%
17%
17%
19%
2000
2002
2004
2006
2008
2010
2012
2014
2015
Year
Successful Challenged Failed
What is Agility? A-gil-i-ty (ә-'ji-lә-tē) Property consisting of quickness,
lightness, and ease of movement; To be very nimble The ability to create and respond to change in order to
profit in a turbulent global business environment The ability to quickly reprioritize use of resources when
requirements, technology, and knowledge shift A very fast response to sudden market changes and
emerging threats by intensive customer interaction Use of evolutionary, incremental, and iterative delivery
to converge on an optimal customer solution Maximizing BUSINESS VALUE with right sized, just-
enough, and just-in-time processes and documentationHighsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
10
What are Agile Methods?
11
People-centric way to create innovative solutions Product-centric alternative to documents/process Market-centric model to maximize business value
Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.orgRico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf
Customer Collaboration
Working Systems & Software
Individuals & Interactions
Responding to Change
valuedmore than
valuedmore than
valuedmore than
valuedmore than
Contracts
Documentation
Processes
Project Plans
Frequent comm. Close proximity Regular meetings
Multiple comm. channels Frequent feedback Relationship strength
Leadership Boundaries Empowerment
Competence Structure Manageability/Motivation
Clear objectives Small/feasible scope Acceptance criteria
Timeboxed iterations Valid operational results Regular cadence/intervals
Org. flexibility Mgt. flexibility Process flexibility
System flexibility Technology flexibility Infrastructure flexibility
Contract compliance Contract deliverables Contract change orders
Lifecycle compliance Process Maturity Level Regulatory compliance
Document deliveries Document comments Document compliance
Cost Compliance Scope Compliance Schedule Compliance
Courage
Agile World View “Agility” has many dimensions other than IT It ranges from leadership to technological agility Today’s focus is on organizational & enterprise agility
Agile Leaders
Agile Organization Change
Agile Acquisition & Contracting
Agile Strategic Planning
Agile Capability Analysis
Agile Program Management
Agile Tech.
Agile Information Systems
Agile Tools
Agile Processes & Practices
Agile Systems Development
Agile Project Management
12
NetworkComputer
Operating SystemMiddlewareApplications
APIsGUI
How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success
13Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional1 2 3 Faster
Early ROI
Lower Costs
Fewer Defects
Manageable Risk
Better Performance
Smaller Attack Surface
Late
No Value
Cost Overruns
Very Poor Quality
Uncontrollable Risk
Slowest Performance
More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction
MINIMIZES MAXIMIZES
JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes
Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
Thousands of TestsContinuously Executed
No More Late BigBang Integration
User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur
14Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
BuildIntegration
Server
VersionControlServer
BuildScripts
UsesWatches
BuildStatus
ProvidesDeveloper A
Developer B
Developer C
CommitsChanges
CommitsChanges
CommitsChanges
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,Efficient, & Repeatable
Constant ReadinessState & CM Control
Lean, Waste Free, Low WIP,No Deadlocked Test Queues
Rapidly & SuccessfullyDev. Complex Systems
Basic Agile Mechanics
15
Methods to “scope” project, product, or system “Key” is smallest possible scope with highest value Reduces cost, risk, time, failure, & tech. obsolescence
Barely Sufficient Design
INCREASES TESTABILITY, QUALITY, RELIABILITY, SECURITY, MORALE, MAINTAINABILITY, & SUCCESS
Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.
MINIMUM
MARKETABLE FEATURE- MMF -
AdvantageDifferenceRevenueProfitSavingsBrandLoyalty
MINIMUMVIABLE PRODUCT
- MVP -
GoalProcessFeaturesPrioritiesStory MapArchitecture
STORY MAPOR IMPACT MAP
- SM or IM -
GoalActors ImpactsDeliverablesMeasuresMilestones
VISIONSTATEMENT
- VS -
For <customer>Who <needs it>The <product> Is a <benefit>That <customer>Unlike <other>Ours <different>
MICRO-SERVICE- MS -
PurposeAutomatedUnique IndependentResilientEcosystemConsumer
16
Capability #1
● Feature 1● Feature 2● Feature 3● Feature 4● Feature 5● Feature 6● Feature 7
Capability #2
● Feature 8● Feature 9● Feature 10● Feature 11● Feature 12● Feature 13● Feature 14
Capability #3
● Feature 15● Feature 16● Feature 17● Feature 18● Feature 19● Feature 20● Feature 21
Capability #4
● Feature 22● Feature 23● Feature 24● Feature 25● Feature 26● Feature 27● Feature 28
Capability #5
● Feature 29● Feature 30● Feature 31● Feature 32● Feature 33● Feature 34● Feature 35
Capability #6
● Feature 36● Feature 37● Feature 38● Feature 39● Feature 40● Feature 41● Feature 42
Capability #7
● Feature 43● Feature 44● Feature 45● Feature 46● Feature 47● Feature 48● Feature 49
1
2 3
4
5 6
7
8 9
10
11 12
13
14 15
16
17 18
19
20 21
Evolving “Unified/Integrated” Enterprise Data Model
“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)
ETL
A A
B C
D E F
G H I J K
A
B C
D E F
A
B C
D E
A
B C
D
A
B C
A
B
“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server
“Leased” DWA/HPC/Cloud Services
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
ETL ETL ETL ETL ETL ETL
Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.
(for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total)
Organize needs into capabilities, features, and stories Prioritize features, group releases, and initiate sprints Develop minimum set of features with highest value
Agile Systems Development
Release
Release
Release
ReleaseMMF- or -MVP
Models of AGILE DEVELOPMENT
17
Agile methods spunoff flexible manufacturing 1990s Extreme Programming (XP) swept the globe by 2002 Today, over 90% of IT projects use Scrum/XP hybrid
Use Cases
Domain Model
Object Oriented
Iterative Dev.
Risk Planning
Info. Radiators
Planning Poker
Product Backlog
Sprint Backlog
2-4 Week Spring
Daily Standup
Sprint Demo
Feasibility
Business Study
Func. Iteration
Design Iteration
Implementation
Testing
Domain Model
Feature List
Object Oriented
Iterative Dev.
Code Inspection
Testing
Release Plans
User Stories
Pair Programmer
Iterative Dev.
Test First Dev.
Onsite Customer
Cockburn, A. (2002). Agile software development. Boston, MA: Addison-Wesley.Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.Stapleton, J. (1997). DSDM: A framework for business centered development. Harlow, England: Addison-Wesley.Palmer, S. R., & Felsing, J. M. (2002). A practical guide to feature driven development. Upper Saddle River, NJ: Prentice-Hall.Beck, K. (2000). Extreme programming explained: Embrace change. Reading, MA: Addison-Wesley.
CRYSTAL METHODS- 1991 -
SCRUM- 1993 -
DSDM- 1993 -
FDD- 1997 -
XP- 1998 -
Reflection W/S Retrospective Quality Control Quality Control Continuous Del.
Basic SCRUM Framework
Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.
Created by Jeff Sutherland at Easel in 1993 Product backlog comprised of prioritized features Iterative sprint-to-sprint, adaptive & emergent model
18
Models of AGILE PROJECT MGT.
19
Dozens of Agile project management models emerged Many stem from principles of Extreme Programming Vision, releases, & iterative development common
Prioritization
Feasibility
Planning
Tracking
Reporting
Review
Visionate
Speculate
Innovate
Re-Evaluate
Disseminate
Terminate
Scoping
Planning
Feasibility
Cyclical Dev.
Checkpoint
Review
Envision
Speculate
Explore
Iterate
Launch
Close
Vision
Roadmap
Release Plan
Sprint Plan
Daily Scrum
Retrospective
Thomsett, R. (2002). Radical project management. Upper Saddle River, NJ: Prentice-Hall.DeCarlo, D. (2004). Extreme project management: Using leadership, principles, and tools to deliver value in the face of volatility. San Francisco, CA: Jossey-Bass.Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.Highsmith, J. A. (2010). Agile project management: Creating innovative products. Boston, MA: Pearson Education.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
RADICAL- 2002 -
EXTREME- 2004 -
ADAPTIVE- 2010 -
AGILE- 2010-
SIMPLIFIED- 2011 -
Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
Created by Mark Layton at PlatinumEdge in 2012 Mix of new product development, XP, and Scrum Simplified codification of XP and Scrum hybrid
20
Simplified AGILE PROJECT MGT.
21
Numerous models of lean development emerging Based on principles of lean thinking & just-in-time Include software, project, & product management
Poppendieck, M., & Poppendieck, T. (2003). Lean software development: An agile toolkit for software development managers. Boston, MA: Addison Wesley.Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. New York, NY: Celeritas.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.Olsen, D. (2015). The lean product playbook: How to innovate with minimum viable products and rapid customer feedback. Hoboken, NJ: John Wiley & Sons.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
LEAN SOFTWARE- 2003 -
LEAN PRINCIPLES- 2009 -
LEAN KANBAN- 2010 -
LEAN PRODUCTS- 2015 -
LEAN ENTERPRISES- 2015 -
Create Value
Eliminate Waste
Amplify Learning
Late Decisions
Deliver Fast
Empower Team
Build-in Integrity
See the Whole
Economic View
Manage Queues
Use Variability
Small Batches
WIP Constraints
Flow Control
Fast Feedback
Decentralize
Visualize
Limit WIP
Manage Flow
Use Policies
Quality Focus
Lead Times
Improvement
Reduce Variation
Target Customer
Market Needs
Market Value
Min. Viability
Prototype
User Experience
Market Testing
Improvement
Measure Risks
Uncertainty
Marketing
Improvement
Value & Flow
Lean Engineering
Experimentation
Bus. Alignment
Models of LEAN METHODS
Basic KANBAN Method
Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Adapted to IT by Dave Anderson in 2006 Activities, buffers, queues, WIP limits, tasks, etc. Lean, JIT pull/demand system leading to high quality
22
23
Numerous models of agile portfolio mgt. emerging Based on lean-kanban, release planning, and Scrum Include organization, program, & project management
Schwaber, K. (2007). The enterprise and scrum. Redmond, WA: Microsoft Press.Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.Larman, C., & Vodde, B. (2008). Scaling lean and agile development: Thinking and organizational tools for large-scale scrum. Boston, MA: Addison-Wesley.Ambler, S. W., & Lines, M. (2012). Disciplined agile delivery: A practitioner's guide to agile software delivery in the enterprise. Boston, MA: Pearson Education.Thompson, K. (2013). cPrime’s R.A.G.E. is unleashed: Agile leaders rejoice! Retrieved March 28, 2014, from http://www.cprime.com/tag/agile-governanceSchwaber, K. (2015). The definitive guide to nexus: The exoskeleton of scaled scrum development. Lexington, MA: Scrum.Org
Models of AGILE PORTFOLIO MGT.
ESCRUM- 2007 -
SAFe- 2007 -
LESS- 2007 -
DAD- 2012 -
RAGE- 2013 -
SPS- 2015 -
Product Mgt
Program Mgt
Project Mgt
Process Mgt
Business Mgt
Market Mgt
Strategic Mgt
Portfolio Mgt
Program Mgt
Team Mgt
Quality Mgt
Delivery Mgt
Business Mgt
Portfolio Mgt
Product Mgt
Area Mgt
Sprint Mgt
Release Mgt
Business Mgt
Portfolio Mgt
Inception
Construction
Iterations
Transition
Business
Governance
Portfolio
Program
Project
Delivery
Product Mgt
Program Mgt
Sprint Mgt
Team Mgt.
Integ Mgt.
Release Mgt
Scaled Agile Framework (SAFE) Created by Dean Leffingwell of Rally in 2007 Knowledge to scale agile practices to enterprise Hybrid of Kanban, XP release planning, and Scrum
24Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.
25
Agile Performance MeasurementW
ork
(Sto
ry, P
oint
, Tas
k)or
Eff
ort
(Wee
k, D
ay, H
our)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Burndown
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Earned Value Management - EVMCPI
SPI
PPC
APC
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Earned Business Value - EBV
What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components
26Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
AGILE TESTING- Early Incremental Testing -
TRADITIONAL TESTING- Late Big Bang Integration Testing -
Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at TimeCode is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports
Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One TimeCode Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports
Code Automatically DeployedLate Defects Freeze Projects
BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated
27Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system
28Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
ALL DEVELOPERS RUN PRIVATE BUILDS
DEVELOPERS COMMIT CODE TO VERSION CONTROL
INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
REPORTS AUTOMATICALLY GENERATED & REVIEWED
Agile testing consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.
29
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
PRACTICES—Continuous Integration
Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline
30Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.
CoQ
• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK
ENTERPRISE—Continuous Delivery
Agile CD consists of seven broad practices Automated acceptance, load, performance, etc. Include packaging, pre-production test, C&A, etc.
31
Practice
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Description
Frequently generating system images for pre-production testing & checkout
Frequently performing automated system & user acceptance testing
Frequently performing automated system load, stress, & capacity testing
Frequently performing automated system user & technical performance testing
Frequently performing automated pre-production tests prior to final deployment
Frequently performing automated system certification & accreditation tests
Frequently generating product images for pre-deployment testing & checkout
Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.
PRACTICES—Continuous Delivery
Created by Patrick Debois of Jedi BVBA in 2007 Collaboration of developers & infrastructure people Goal to automate the deployment to end-user devices
32Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
GLOBAL—Development Operations
Agile DevOps consists of seven broad practices Automated sys admin, CM, building, monitor, etc. Include virtualization, containerize, deployment, etc.
33
Practice
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerize
Deployment
Monitor & Supp
Description
Frequently performing automated system administration tasks, i.e., scripting
Frequently performing automated infrastructure config. mgt./version control
Frequently performing automated system and server host builds and config.
Frequently performing automated system, server, & net virtualization services
Frequently performing automated software and Microservices containerization
Frequently generating final end-user system & software images for distribution
Frequently performing automated metrics collection & deployment monitoring
Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.
GLOBAL—Development Operations
Agile methods are based on traditional measures Story points, velocity, and burndown basic metrics Experts use Agile EVM, test, ROI & portfolio metrics
34Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
AGILE METRICS1. Agile CODE Metrics2. Agile PROJECT Metrics3. Agile TRACKING Metrics4. Agile TESTING Metrics5. Agile VALUE Metrics6. Agile HEALTH Metrics7. Agile PORTFOLIO Metrics
1. Agile CODE Metrics Code Size Code Complexity Object Oriented Code Coverage Code Defects Relational Design
2. Agile PROJECT Metrics Software Size Software Productivity Software Effort Software Quality Software Schedule Software Success
3. Agile TRACKING Metrics Story Points Sprint Burndown Release Burndown Velocity Feature Progress Agile Earned Value
4. Agile TESTING Metrics Test Coverage Test Automation Integration Builds Running Tested Features DevOps Automation Deployment Frequency
7. Agile PORTFOLIO Metrics Portfolio Kanban Epic Progress Portfolio Radar Release Train Radar Lean Portfolio Metrics Enterprise Scorecard
6. Agile HEALTH Metrics Teamwork Quality Collaboration Quality Agile Process Maturity Agile Adoption Rate Degree of Agility Product Flexibility
5. Agile VALUE Metrics Total Lifecycle Costs Total Lifecycle Benefits Benefit to Cost Ratio Return on Investment Net Present Value Real Options Analysis
Agile Testing Metrics—Taxonomy
35
METRIC DESCRIPTION
TEST COVERAGE Percent or degree to which software source code is tested
TEST AUTOMATION Ratio or degree to which software tests are automated
INTEGRATION BUILDS Frequency of automated software builds and integrations
RUNNING TESTED FEATURES Number of completed and tested features or user stories
DEVOPS AUTOMATION Ratio or degree to which deployments are automated
DEPLOYMENT FREQUENCY Frequency of automated software deployments or deliveries
Software test automation emerged during the 1970s Reached their height in personal computer (PC) era Most are FOSS and used by successful agile teams
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing Metrics—Definitions
Agile Testing Metrics—Example
36Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
37
Traditional vs. Agile Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Agile Testing—Workflow
Fewer integrations leave in higher bug counts Frequent, early integrations eliminate most defects Goal is to have as many early integrations as possible
38Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
Number ofIntegrations
Less Defects•More Integrations•Early IntegrationsMore Defects
•Few Integrations•Late Integrations
Agile Testing—Economic Drivers
Tests Hour Day Week Month Quarter 6 Months YearOne 6 48 240 1,040 3,120 6,240 12,480
Three 18 144 720 3,120 9,360 18,720 37,440Six 36 288 1,440 6,240 18,720 37,440 74,880
Twelve 72 576 2,880 12,480 37,440 74,880 149,760Eighteen 108 864 4,320 18,720 56,160 112,320 224,640
Activity Def CoQ DevO ps Economics H ours RO I"AGILE" Testing 100 0.1 100 D efects x 70% Eff. x 0.1 Hours 7 8,100%
Software Inspections 30 1 30 D efects x 70% Eff. x 1 Hours 21 2,700%
"Traditional" Testing 9 10 9 D efects x 70% E ff. x 10 Hours 63 900%
Manual Debugging 3 100 2.7 D efects x 70% Eff. x 100 Hours 189 300%
Operations & Maintenance 0.81 1,000 0.81 D efects x 70% E ff. x 1,000 Hours 567 n/a
Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes
39Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Agile Testing—Economics
Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often
40Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Agile Testing—Cost of Quality
Agile Cost & Benefit Analysis Costs based on avg. productivity and quality Productivity ranged from 4.7 to 5.9 LOC an hour Costs were $588,202 and benefits were $3,930,631
41Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation. Ft. Lauderdale, FL: J. Ross Publishing.
d1 = [ln(Benefits Costs) + (Rate + 0.5 Risk2) Years] Risk Years, d2 = d1 Risk Years
5
1i
Benefits of Agile Methods Analysis of 23 agile vs. 7,500 traditional projects Agile projects are 54% better than traditional ones Agile has lower costs (61%) and fewer defects (93%)
Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.
Project Cost in Millions $
0.75
1.50
2.25
3.00
2.8
1.1
Before Agile
After Agile
61%LowerCost
Total Staffing
18
11
Before Agile
After Agile
39%LessStaff
5
10
15
20
Delivery Time in Months
5
10
15
20
18
13.5
Before Agile
After Agile
24%Faster
Cumulative Defects
625
1250
1875
2500
2270
381
Before Agile
After Agile
93%Less
Defects
42
Agile vs. Traditional Success Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less often
Standish Group. (2012). Chaos manifesto. Boston, MA: Author.
43
Agile Traditional
Success42%
Failed9%
Challenged49%
Success14%
Failed29%
Challenged57%
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing
44
Agile Testing—CI Statistics
45
Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.
TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 x
Regression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
Agile Testing—CD Statistics
46Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
62x FasterU.S. DoD
IT Project
3,645x FasterU.S. DoD
IT Project
Agile Testing—DevOps Statistics Goal of continuous delivery is releases vs. build/tests Market-driven releases creates rapid business value Assembla went from 2 to 45 monthly releases w/CD
Google early adopter of agile methods and Scrum Google also uses agile testing at enterprise scale 15,000 developers run 120 million tests per day
47Micco, J. (2013). Continuous integration at google scale. Eclipse Con, Boston, MA.Whittaker, J., Arbon, J., & Carollo, J. (2012). How google tests software. Upper Saddle River, NJ: Pearson Education.
440 billion unique users run 37 trillion searches each year Single monolithic code tree with mixed language code Submissions at head – One branch – All from source 20+ code changes/minute – 50% code change/month 5,500+ submissions/day – 120 million tests per day 80,000 builds per day – 20 million builds per year Auto code inspections – For low defect density 10X programming productivity improvement $150 million in annual labor savings (ROI as a result)
Agile Testing—Google Statistics
Amazon adopted agile in 1999 and Scrum in 2004 Using enterprise-scale continuous delivery by 2010 30,000+ developers deploy over 8,600 releases a day
48Atlas, A. (2009). Accidental adoption: The story of scrum at amazon.com. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 135-140.Jenkins, J. (2011). Velocity culture at amazon.com. Proceedings of the Velocity 2011 Conference, Santa Clara, California, USA.Elisha, S. (2013). Continuous deployment with amazon web services. Proceedings of the AWS Summit 2013, Sydney, New South Wales, Australia.
Software deployment every 11.6 seconds (as of 2011) 24,828 to 86,320 releases per Iteration 161,379 to 561,080 releases per Quarter 645,517 to 2,244,320 releases per Year
Automatic, split-second roll-forward & backward 75-90% reduction in release-caused outages (0.001%) Millions of times faster (than traditional methods) 4,357,241 to 15,149,160 per traditional release
Thousands of times faster (than manual agility) 161,379 to 561,080 per Scrum/SAFe release
Used agile methods long before U.S. government (1999)
Agile Testing—Amazon Statistics
Activity Def CoQ DevOps Economics Hours ROIDevelopment Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%
Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%
Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%
Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%
"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%
Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%
Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a
49
Agile testing is orders-of-magnitude more efficient Based on millions of automated tests run in seconds One-touch auto-delivery to billions of global end-users
Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com
Agile Testing—DevOps CoQ
50
CoQ increases with each life cycle phase Inspection CoQ is best traditional methods offer DevOps yields orders-of-magnitude smaller CoQ
Ambler, S. (2009). Examining the agile cost of change curve. Retrieved March 20, 2012 from http://www.agilemodeling.com
Operations & Maintenance
Manual Debugging
“Traditional” Testing
Software Inspections
Continuous Integration
Continuous Delivery
Development Operations
Agile Testing—DevOps CoQ Cont’d
Microsoft created software security life cycle in 2002 Waterfall approach tailored for Scrum sprints in 2009 Uses security req, threat modeling & security testing
51
Microsoft. (2011). Security development lifecycle: SDL Process Guidance (Version 5.1). Redmond, WA: Author.Microsoft. (2010). Security development lifecycle: Simplified implementation of the microsoft SDL. Redmond, WA: Author.Microsoft. (2009). Security development lifecycle: Security development lifecycle for agile development (Version 1.0). Redmond, WA: Author.Bidstrup, E., & Kowalczyk, E. C. (2005). Security development lifecycle. Changing the software development process to build in security from the start. Security Summit West.
SEE DETAILED - SECURITY LIFE CYCLE STEPShttp://davidfrico.com/agile-security-lifecycle.txt
Agile Testing—Security Life Cycle
Enables enterprises to be flexible but disciplined Allows enterprises to distribute project work teams Ensures distributed project teams are collaborating
52
Agile Tools“Across the Life Cycle”
Project Management
RequirementsDOORSRequisite ProSLATE
DesignRhapsodyTelelogic System ArchitectRational System Architect
CodingEclipseVisual StudioSun Studio
TestingJUnitNUnitXunitCPPUnit
GtestFitFitnesseSelenium
Quality AssuranceCheckStylePMDEMMAJdependCoberturaGcov
Configuration MgtSubversion (SVN)Concurrent Versions Sys.ClearCase
Build AutomationAntNAntMavenMake
Continuous Integ.Cruise ControlHudsonBuildBot
CollaborationWebExSkypeMeetMeWimba
WikiMediaWikiTracWikiPhpWiki
DocumentationNDocJavadocDoxygeniText
Version OneRallyScrum WorksVSTS
Agile TeamAgile EnterpriseScope ManagerStory Studio
XP Plan ItIterateXP TrackerAgilo
XP CGIXP WebXplannerIce Scrum
Project CardsTarget ProcessXtreme PlannerTeam System
CommunityEnterpriseMingleHansoft
There are literally hundreds of agile testing tools There are tools for building, testing, and deployment Integration tools monitor repositories and initiate tests
53
Agile Tools“In-Depth Test Automation”
Smart, J. (2009). Automated deployment with maven and friends: Going the whole nine yards. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
Simple example of a DevOps reference architecture Includes CM, continuous integration, & deployment Code automatically built/tested/deployed to users
54
Agile Tools“Simple DevOps Automation”
Morris, B., & Cassatt, C. (2015). Devops for the rest of us. Proceedings of the Agile DC Conference, Washington, DC, USA.Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
55
Agile Tools“Periodic Table of DevOps Automation”
XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.
56Holler, R. (2015). Ninth annual state of agile survey: State of agile development. Atlanta, GA: VersionOne.
VersionOne found 94% using agile methods today Most are using Scrum with several key XP practices Lean-Kanban is a rising practice with a 31% adoption
ContinuousIntegration
●●
●
●
●
●
●
●
●
●●
●
●
Agile Testing—Adoption Statistics
Agile test use is low in spite of its age, i.e., 15 years Many do not understand its utter simplicity and power Failure to use agile testing undermines project success
57Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.
Agile PracticesRetrospectives
Refactoring
Done Definition
Test Tools
Test Driven Dev.
CM Tools
Simplicity
Pair Programming
Technical Debt
Agile Testing 13%
Continuous Integrations
Weekly
Daily
2-3 TimesPer Day
Never
2-3Times
PerIteration
Agile Testing—Usage Statistics
Agile teams don’t often use TDD, CI, CD & DevOps Implement independent test teams after Sprints done Sprint Waterfalling, Scrummerfalling, & Wagile result
58Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.
Incorrect• Phased Testing• Separate Teams• Delayed Testing
Correct• Integrated Testing• Integrated Teams• Continuous Testing
Agile Testing—Anti-Patterns
Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention
59Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
MICRO ADJUSTMENTS- Focused Impact Tuning-
MACRO ADJUSTMENTS- Wide Impact Tuning-
Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test LibrariesReduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware
In-Memory CompilationParallelize Test RunsPre-Install Test LibrariesRemove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process
Agile Testing—Scaling Practices
Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing
60
Technical BarriersOrganizational BarriersDevelopers don’t want to test
· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing
Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture
··
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing—Common Barriers
Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project
61Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
What’s the Bottom Line?“Agile Testing Done Early & Often”
Agile TestingTraditional TestingDramatically reduces risks
· Automates manual processes· Instant verification & validation· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability
Late defect discovery· Low quality software· Poor project visibility· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure
··
MIT’s Ten (10) LAWS OF SIMPLICITY One must think and act small to accomplish big things Slow down to speed up, speed up ‘til wheels come off Scaling up lowers productivity, quality, & business value
62Maeda, J. (2006). The laws of simplicity: Design, technology, business, and life. Cambridge, MA: MIT Press.
REDUCE - The simplest way to achieve simplicity is through thoughtful reduction.
ORGANIZE - Organization makes a system of many appear fewer.
TIME - Savings in time feel like simplicity.
LEARN - Knowledge makes everything simpler.
DIFFERENCES - Simplicity and complexity need each other.
CONTEXT - What lies in the periphery of simplicity is definitely not peripheral.
EMOTION - More emotions are better than less.
TRUST - In simplicity we trust.
FAILURE - Some things can never be made simple.
THE ONE - Simplicity is about subtracting the obvious and adding the meaningful.
Conclusion Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality
63Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdfRico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdfRico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance
Dave’s PROFESSIONAL CAPABILITIES
64
SoftwareQuality
Mgt.
TechnicalProject
Mgt.
SoftwareDevelopment
Methods
Strategy &Roadmapping
SystemsEngineering
Cost Estimates& Scheduling
Acquisition &Contracting
OrganizationChange
Lean, Kanban,& Six Sigma
Modeling &Simulations
Big Data,Cloud, NoSQL
WorkflowAutomation
Metrics,Models, & SPC
BPR, IDEF0,& DoDAF
DoD 5000,TRA, & SRA
PSP, TSP, &Code Reviews
CMMI &ISO 9001
InnovationManagement
Statistics, CFA,EFA, & SEM
ResearchMethods
EvolutionaryDesign
Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOpsSec, etc.
STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.
● Data mining. Metrics, benchmarks, & performance.● Simplification. Refactoring, refinement, & streamlining.● Assessments. Audits, reviews, appraisals, & risk analysis.● Coaching. Diagnosing, debugging, & restarting stalled projects.● Business cases. Cost, benefit, & return-on-investment (ROI) analysis.● Communications. Executive summaries, white papers, & lightning talks.● Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.
PMP, CSEP,FCP, FCT, ACP,CSM, SAFE, &
DEVOPS
34+ YEARSIN IT
INDUSTRY
Books on Agile Testing Thousands of textbooks on agile methods Include requirements, design, coding, test, etc. Continuous Integration, Delivery, & DevOps best
65
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Gregory, J., & Crispin, L. (2015). More agile testing: Learning journeys for the whole team. Upper Saddle River, NJ: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery: Reliable software releases through build, test, and deployment automation. Boston, MA: Pearson Education.
Books on ROI of SW Methods Guides to software methods for business leaders Communicates the business value of IT approaches Rosetta stones to unlocking ROI of software methods
http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)
66