#

Mark LambertVP Product Management and Support

How to Avoid Continuously Delivering Faulty Software

#

Software Development ChallengesSoftware Development Challenges

APIs drive interconnectivity across the expanded internet

Compliance with regulatory, industry and internal standards

SDLC Speed will be the difference between a first mover vs follower

#

Software Drives InnovationSoftware Drives Innovation

The Cost of Quality associated with software has shifted dramatically

Software has shifted from process enabler to business differentiator

Quality drives brand loyalty

#

The Cost of Software QualityThe Cost of Software Quality

• After a rash of software failures and security breaches left Sony’s gaming services down for weeks, analysts called for the ousting of the Sony CEO.

• Cumulative Loss = $18B

Mar

ket

Val

ue

Event 15d 30d

-22% -33% -30%-19% -11% -12%

#

• Constant Trade-offs that have business impact

SDLC - The Era of Acceleration SDLC - The Era of Acceleration

Time

Quality

Scope

#

• Continuous testing accelerates the SDLC bymanaging quality expectations and actionable tasks

From Automated to Continuous

Expectations

Policy Management

Development

Defect Prevention

Development

Development Testing

Static Analysis

Unit/Component

Peer Review

Automated Tests

Integration Testing

API/Service Tests

Smoke Test

Security Tests

Automated Tests

System Testing

Functional Tests

Scenario Tests

Performance Tests

ContiniousBuild

Remediation Tasks

Go

Release

No Go

???

Service Virtualization – Test Environment Access

#

Static Code Analysis

Pattern-Based Static Analysis

Prevention technique

Analyzes code structure (parse

tree) to apply best practices

Flow-Based Static Analysis

Detection technique

Analyzes code flow to determine

“dangerous paths”

Metric Threshold Analysis

Advisory technique

Finds complex/hard-to-test code prone to

errors

#

• Well understood often under valued• Define the goal of the analysis and the Policy for

compliance• Focus on reduction of business risk not pursuit of

perfection• Start small to promote adoption and monitor for areas of

improvement

Static Code Analysis

#

Peer Code Reviews

• Highly valuable in finding REAL bugs– Algorithms/Design

• Use carefully– Only apply after Static

Code Analysis– Only apply where

there is Business RISK

Image: http://www.jasonawesome.com/2010/06/01/executing-a-php-code-review/

#

• Unit Testing– Developer focuses on the code– Typically not true Unit Test– Code needs to be built to be testable

• Where is the ROI?– Did we design it properly

• How much is enough?– Code Coverage + Peer Review

Unit vs. Functional Testing

#

• Functional Testing– QA focused on the user-story/function

• Where is the ROI?– Does it function correctly– Did we break functionality

• How much is enough?– User-story coverage

• Assoc. code coverage provides additional insight

Unit vs. Functional Testing

#

• Ad-hock/Unstructured Testing of functional areas

• Important part of QA/feedback process

• Requires traceability to user-stories and code

• Should be ‘reinforced’ with automated tests

Explorative Testing

#

• Limitations– Often at the end of the cycle

• Wait until the whole system is ready– Requires specialized skills and specialized tools

• Often not “real tests”– Too late for cost effective remediation

• “Shift Left” Performance and Security– Reuse automated functional tests and tooling– Eliminated the system constraints … Service Virtualization …

Performance/Security Testing

#

• Complexity is a Barrier to Innovation– Accessible– Stable – Controllable

• Constrained Testing

Service Virtualization

3rd Party System

Evolving Component

Mainframe

Scheduled Access

#

• Emulates dependencies for the Test Environment– Reduces the complexity for early stage testing– Increases predictability

• Enables “Test Anytime, Anywhere, Anyway”– Automated Provisioning for different use-cases– Automated Test Data Management/Simulation

• Does not eliminate the need for System/Integration Testing

Service Virtualization

#

Continuous Test Characteristics

• Logically componentized• Correlated with business

requirements • Incremental, Repeatable• Maintainable,

Deterministic• Process is prescriptive

based on results

Continuous Testing

Policy

Traceability

Analysis Risk Assessment

Environment Access

Optimization

#

Development Testing Platform

• Centralize and Automated “Quality Hub”

• Provide Controls and visibility onto variable and ad-hoc usage of quality tools (incl. open source)

• Enables centralized policy to drive consistent results of the SDLC practices

DTP

Source Control

Defects

Require-ments

Code Review

Static Analysis

MetricsFlow Analysis

Unit Testing

Coverage

Functional Testing

Load Testing

#

Workflow drives improvement

• Aggregation of objective SDLC data transformed into actionable information

• Identify, and prevent, potential defects to reduce project risks

• Developer workflow driven from the Developer

Code Analysis

Data Aggregation

Post Analysis Analysis

(PIE)

Reporting and

Prioritization

Download to IDE

Developer Remediation

Source Check-in

DTP

#

• Real-time feedback on compliance and certification with industry, regulatory or standards initiatives during active development.

Visibility for Compliance

#

• Bridge the gap between technical findings and business impact– Aggregation of deep SDLC data into informative dashboards that.

Provide Clarity on Risk

#

Demonstration

#

1. Define Business Expectations in a Policy

2. Automate Key Software Quality Practices1. Code Analysis

2. Peer Review

3. Automated Testing with Traceability

3. Apply Continuously and with a Workflow for remediation

4. Translate to Business Impact and Monitor for improvements

How to Avoid Continuously Delivering Faulty Software

##

Thank you!Mark Lambert

Mark.Lambert@parasoft.com

@mark_l_lambert