Corey Vaudo

Conrad Rybka

UVA Software Testing - Fall 2019

How quality fits in every step of the software development process

Quality Everywhere

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Agenda

Who We Are Defining Quality Quality Everywhere Q & A

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

3

Who We Are

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

4

Mastercard Data & Services

Mission

Help drive evidence-based business decisions

Clients

Deployed globally by over 300 organizations

Expertise

Combine business strategy, math, and large databases to inform decisions

Product

Cloud based software platform utilizing same underlying technology across industries

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

5

Enabling business experimentation

Test & Learn

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

6

Promotion Planning – is my promotion driving traffic or just eroding margin?

Menu Development – what entrees should I remove from my menu? What should I add?

Market Basket Profiling – How do different customers react to a promotion? How can we target promotions by customer segment?

Customer Incentive Programs – what rate should I offer on a CD to a new customer?

Network Planning – where should we build the next Holiday Inn?

Online-to-store Advertising - how much does online advertising affect my in-store sales? Do online sales cannibalize in-store sales?

Business experiment examples

Test & Learn

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

7

Technology

Big Data

• Host over 500 individual SQLServerdatabases

• Host over 1 PB of data

• Conduct “sku” level analysis for the world’s largest retailers

Analytics

• Hold numerous patents for advanced analytic techniques

• Utilize both leading statistical packages and develop in-house algorithms

Web Development

• Host software entirely over the web utilizing latest front-end technologies (e.g. Redux, React, etc)

CI / CD

• Deploy code multiple times per day

• Run tens of thousands of tests daily

• Utilize git, Jenkins, nUnit, Chef, Selenium & numerous other world-class “testing & deployment” technologies

Our approach requires expertise in numerous areas

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

8

Defining Quality

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Stereotypical approach to quality

9

Product Management team develops “requirements”

Engineers build software to “spec”

QA team confirms software is high quality

Software delivered to users

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

10

Continuous IntegrationAnalytic Accuracy

LicensingAlerting

Usability

Reliability

Regression Testing

Dogfooding

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

11

Quality Everywhere

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

12

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

13

Prototyping

Quality in the requirements process

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

14

Usability Sessions

Quality in the requirements process

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

15

Requirements Testing

Quality in the requirements process

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

16

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

17

Pull Requests and Code Reviews

Quality is shared amongst all developers

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

18

Pair Programming

Quality is shared amongst all developers

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

19

Test Driven Development

Quality is shared amongst all developers

Add Test

Watch Test Fail

Write CodeRun Tests

Refactor

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

20

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

21

Continuous integration tools ensure that all changes are merged and tested together frequently

Code is “integrated” with every commit

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

22

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

23

When you fix one bug, you introduce several newer bugs

Automated regression testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

24

Manually test only once, test automatically forever

Automated regression testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

25

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

Security Testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

26

Static and dynamic code analysis

Vulnerability scanning

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

27

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

Performance Testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

28

Front-end page load tests

Performance testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

29

Analytic code performance monitoring

Performance testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

30

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

31

Server consistency is a key driver of quality

Quality through configuration management

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

32

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

Integration Testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

33

Test against live versions of the application

Front-end integration tests

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

34

Can the application perform under significant load

Load testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

35

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

Monitoring and Alerting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

36

Extensive monitoring of production systems minimizes downtime

Monitoring and alerting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

37

RequirementsGathering

Development

ContinuousIntegration

Regression Testing

Servers

Analytic and Front-endTesting

Usage

User Feedback &Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

38

Extensive test planning and review leads to more effective testing

Front-end & analytic testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

39

Testing blitzes allow for testing from multiple perspectives

Front-end & analytic testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

40

Analytic validation is essential to providing a trustworthy tool

Front-end & analytic testing

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

41

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

42

Multiple deployment environments allow for internal user feedback

Internal usage

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

Quality is built-in everywhere throughout the development process

43

Requirements Gathering

Development

Continuous Integration

Regression Testing

Servers

Analytic and Front-end Testing

Usage

User Feedback & Reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

44

Monitoring usage and collecting feedback helps close the loop between development teams and their clients

User feedback and reporting

©2

01

8 M

aste

rca

rd.

Pro

pri

eta

ry a

nd

Co

nfid

en

tia

l.

45

Q & A