how to achieve security, reliability, and productivity in less time

37
1 © 2017 Rogue Wave Software, Inc. All Rights Reserved. 1 Confronting the mission-critical software testing challenge Episode 1: How to achieve security, reliability, and productivity in less time Rod Cope CTO

Upload: rogue-wave-software

Post on 21-Feb-2017

98 views

Category:

Software


1 download

TRANSCRIPT

Page 1: How to achieve security, reliability, and productivity in less time

1© 2017 Rogue Wave Software, Inc. All Rights Reserved.

1

Confronting the mission-critical software testing challengeEpisode 1:

How to achieve security, reliability, and productivity in less timeRod CopeCTO

Page 2: How to achieve security, reliability, and productivity in less time

2© 2017 Rogue Wave Software, Inc. All Rights Reserved.

2

Presenter

Rod CopeCTORogue Wave [email protected]: @RodCope

Page 3: How to achieve security, reliability, and productivity in less time

3© 2017 Rogue Wave Software, Inc. All Rights Reserved.

3

1. A brief history of testing2. Code security3. Software reliability4. Pulling it together5. Q&A

Agenda

Page 4: How to achieve security, reliability, and productivity in less time

4© 2017 Rogue Wave Software, Inc. All Rights Reserved.

4

What is mission-critical?

Page 5: How to achieve security, reliability, and productivity in less time

5© 2017 Rogue Wave Software, Inc. All Rights Reserved.

5

A brief history of testing

Page 6: How to achieve security, reliability, and productivity in less time

6© 2017 Rogue Wave Software, Inc. All Rights Reserved.

6

The evolution of testing

1970s – 80s Debugging == testing

All I need is unit testing

How did we survive without automated testing?

DevOps is awesome!

1990s

2010s

2000s

Page 7: How to achieve security, reliability, and productivity in less time

7© 2017 Rogue Wave Software, Inc. All Rights Reserved.

7

Challenges with different methodsAdvantages Disadvantages

Debugging/printfs

• Immediate• Minimal set up

• Limited view of system• Limited tests• Doesn’t scale across

code/team sizeUnit testing • Close to code

• A form of documentation

• Limited view of system• Limited tests• Cumbersome for single

developer to set upBasic automated testing

• Consistency and repeatability

• Speed• Frees developer time

• Can be slow to run• Can be slow to update

DevOps/CI testing

• Consistency and repeatability

• Scalable & fast• Frees developer time

• Initial set-up costs• Only effective for larger

teams

Page 8: How to achieve security, reliability, and productivity in less time

8© 2017 Rogue Wave Software, Inc. All Rights Reserved.

8

Challenges with different methodsAdvantages Disadvantages

Debugging/printfs

• Immediate• Minimal set up

• Limited view of system• Limited tests• Doesn’t scale across

code/team sizeUnit testing • Close to code

• A form of documentation

• Limited view of system• Limited tests• Cumbersome for single

developer to set upBasic automated testing

• Consistency and repeatability

• Speed• Frees developer time

• Can be slow to run• Can be slow to update

DevOps/CI testing

• Consistency and repeatability

• Scalable & fast• Frees developer time

• Initial set-up costs• Only effective for larger

teams

Page 9: How to achieve security, reliability, and productivity in less time

9© 2017 Rogue Wave Software, Inc. All Rights Reserved.

9

Challenges with different methodsAdvantages Disadvantages

Debugging/printfs

• Immediate• Minimal set up

• Limited view of system• Limited tests• Doesn’t scale across

code/team sizeUnit testing • Close to code

• A form of documentation

• Limited view of system• Limited tests• Cumbersome for single

developer to set upBasic automated testing

• Consistency and repeatability

• Speed• Frees developer time

• Can be slow to run• Can be slow to update

DevOps/CI testing

• Consistency and repeatability

• Scalable & fast• Frees developer time

• Initial set-up costs• Only effective for larger

teams

Page 10: How to achieve security, reliability, and productivity in less time

10© 2017 Rogue Wave Software, Inc. All Rights Reserved.

10

Challenges with different methodsAdvantages Disadvantages

Debugging/printfs

• Immediate• Minimal set up

• Limited view of system• Limited tests• Doesn’t scale across

code/team sizeUnit testing • Close to code

• A form of documentation

• Limited view of system• Limited tests• Cumbersome for single

developer to set upBasic automated testing

• Consistency and repeatability

• Speed• Frees developer time

• Can be slow to run• Can be slow to update

DevOps/CI testing

• Consistency and repeatability

• Scalable & fast• Frees developer time

• Initial set-up costs• Only effective for larger

teams

Page 11: How to achieve security, reliability, and productivity in less time

11© 2017 Rogue Wave Software, Inc. All Rights Reserved.

11

All have sources of risk

• Human error• Software issues• Hardware

issues

And challenges:• Demands for shorter release times• Increasing feature complexity• Requirements for standards

compliance• Increasing open source use

Page 12: How to achieve security, reliability, and productivity in less time

12© 2017 Rogue Wave Software, Inc. All Rights Reserved.

12

Poll #1What is the primary method you use to test code?• Code reviews• Unit tests• Manual tests at build time• Automated tests at build time• Automated testing using CI tools

Page 13: How to achieve security, reliability, and productivity in less time

13© 2017 Rogue Wave Software, Inc. All Rights Reserved.

13

Code security

Page 14: How to achieve security, reliability, and productivity in less time

14© 2017 Rogue Wave Software, Inc. All Rights Reserved.

14

Changing security landscape

More complex software running inside systems

Multiple sources of software being integrated

Software has to run for many years

This requires a very significant security, safety, & functional verification process

Harder to secure code

Page 15: How to achieve security, reliability, and productivity in less time

15© 2017 Rogue Wave Software, Inc. All Rights Reserved.

15

Some research

"Security is not considered important"

"Security takes too much time"

"I feel pressured to complete development"

0% 5% 10% 15% 20% 25%

22%

22%

24%

Why are companies not putting more emphasis on security in

their applications?

Yes

No

0% 10% 20% 30% 40% 50%

49%

51%

Is security a priority for your company?

2015 Survey of Automakers and SuppliersPonemon Institute / Rogue Wave Software / Security

Innovation

Page 16: How to achieve security, reliability, and productivity in less time

16© 2017 Rogue Wave Software, Inc. All Rights Reserved.

16

One of the top flaws in the 2015

National Vulnerability

Database

Example: Memory buffer problems

CWE-119: Software can read or write to locations outside of the boundaries of the memory buffer

• Not checking size of input on copy• Bug allowing writing to arbitrary

locations• Out-of-bounds read• Pointers outside expected range• Untrusted pointer dereference• Uninitialized pointers• Expired pointer references• Access of memory beyond buffer end

Page 17: How to achieve security, reliability, and productivity in less time

17© 2017 Rogue Wave Software, Inc. All Rights Reserved.

17

Real vulnerability: GNU libc

CVE-2015-1472https://sourceware.org/ml/libc-alpha/2015-02/msg00119.html

• Under certain conditions wscanf can allocate too little memory for the to-be-scanned arguments and overflow the allocated buffer.

• Theoretically, any Linux machine connected to the internet, using this version, is at risk

Page 18: How to achieve security, reliability, and productivity in less time

18© 2017 Rogue Wave Software, Inc. All Rights Reserved.

18

GNU libc example: fail

Page 19: How to achieve security, reliability, and productivity in less time

19© 2017 Rogue Wave Software, Inc. All Rights Reserved.

19

GNU libc example: fix

Page 20: How to achieve security, reliability, and productivity in less time

20© 2017 Rogue Wave Software, Inc. All Rights Reserved.

20

Top four best security practices

• Numeric errors• Code injection• Improper input

validation• Memory buffer

problems

• Numeric errors• Cryptographic issues• Code injection• Memory buffer problems

• Numeric errors• Cryptographic

issues• Code injection• Resource

management errors

• Numeric errors• Resource management errors• Improper access control• Improper input validation

Clean design

Methodical process

Good tools

Careful analysis

Page 21: How to achieve security, reliability, and productivity in less time

21© 2017 Rogue Wave Software, Inc. All Rights Reserved.

21

Poll #2How much time do developers in your company spend on security (as a percentage of work time)?• 0%• 1 – 25%• 26 - 50%• 51 – 75%• 76 – 100%

Page 22: How to achieve security, reliability, and productivity in less time

22© 2017 Rogue Wave Software, Inc. All Rights Reserved.

22

Software reliability

Page 23: How to achieve security, reliability, and productivity in less time

23© 2017 Rogue Wave Software, Inc. All Rights Reserved.

23

Why is reliability important?

May 2015• Boeing 787 Dreamliner had software bug which

caused “total loss of electrical power” after 248 days

December 2015• A software error which calculates prison sentences caused more

than 3,200 US prisoners to be released 49 days early on average

Page 24: How to achieve security, reliability, and productivity in less time

24© 2017 Rogue Wave Software, Inc. All Rights Reserved.

24

January 6, 2016

• NEST ‘smart’ thermostat software update caused complete battery drain, shutting off heat during January

• Matt Rogers, NEST co-founder & VP Eng:”the bug took a few weeks to show up”

• 2.5 million smart thermostats in U.S. alone

Page 25: How to achieve security, reliability, and productivity in less time

25© 2017 Rogue Wave Software, Inc. All Rights Reserved.

25

Key industry standards

The argument for standards compliance

• Re-use the expert research of others• Complements existing testing approaches• Recognizable by customers• May already by a requirement

Significantly reduces the cost of producing reliable software

Security:

Page 26: How to achieve security, reliability, and productivity in less time

26© 2017 Rogue Wave Software, Inc. All Rights Reserved.

26

MISRA C example

a |= 256;b |= 128; c |= 064;

Sets bit 8 of variable a(256 decimal = 0100000000 binary)

Sets bit 7 of variable b(128 decimal = 0010000000 binary)

Is bit 6 set?(64 decimal = 000100000 binary)

Rule 7.1: Octal constants (other than zero) and octal escape sequences shall not be used.

• No, because in C, any constant that begins with 0 is interpreted as an octal number.

• So c is set to the wrong value!

Page 27: How to achieve security, reliability, and productivity in less time

27© 2017 Rogue Wave Software, Inc. All Rights Reserved.

27

Pulling it together

Page 28: How to achieve security, reliability, and productivity in less time

28© 2017 Rogue Wave Software, Inc. All Rights Reserved.

28

Why Agile?

Agile increasing

Page 29: How to achieve security, reliability, and productivity in less time

29© 2017 Rogue Wave Software, Inc. All Rights Reserved.

29

People over processes

Collaborate to build trust and foster change

Set expectations clearly

Test and measure

Share successes

Enable with tools

Pick the right

artifacts

Choose what to keep/throw

away

Lessons learned

Page 30: How to achieve security, reliability, and productivity in less time

30© 2017 Rogue Wave Software, Inc. All Rights Reserved.

30

Continuous testing• Check for security issues• Measure conformance to

standards• Examples of CI systems:

TeamCity, Jenkins• Examples of test tool:

static code analysis

Automate testing

AcceptCheck in

Dev 1

Check inDev 2

Check inDev 3 Release

ChangeAdjust and Track

FeedbackReview

Next Iteration

No!

Release to

Market

Test

TestTest

Yes!

Page 31: How to achieve security, reliability, and productivity in less time

31© 2017 Rogue Wave Software, Inc. All Rights Reserved.

31

Keys to successful CITo work in a true CI environment test tools must be designed to be:

Automated

Fast(er)

Scalable

RelevantTo reduce feedback time, only changed code should be tested (including regression)

By requiring minimal resources & deploying across multiple agents

By reporting only the information that is required for the given context (example: only the diffs since the last build / build X)

Supporting the most important CI build management systems

Page 32: How to achieve security, reliability, and productivity in less time

32© 2017 Rogue Wave Software, Inc. All Rights Reserved.

32

Summary

• Identify and prevent vulnerabilities before release

Security: Clean design, methodical process, careful analysis, good tools

• MISRA, OWASP, ISO 26262Reliability: Adopt proven standards

• Jenkins, static code analysis

Automate with tools that are fast, scalable, and relevant

1

2

3

Page 33: How to achieve security, reliability, and productivity in less time

33© 2017 Rogue Wave Software, Inc. All Rights Reserved.

33

Q & A

Page 34: How to achieve security, reliability, and productivity in less time

34© 2017 Rogue Wave Software, Inc. All Rights Reserved.

34

Follow up

Free white paper:

Fitting static code analysis into continuous integrationwww.roguewave.com/resources/white-papers/static-code-analysis-into-continuous-integration

Page 35: How to achieve security, reliability, and productivity in less time

35© 2017 Rogue Wave Software, Inc. All Rights Reserved.

35

Missed this webinar? Watch it on-demand

How to achieve security, reliability, and productivity in

less time

Watch now.

Page 36: How to achieve security, reliability, and productivity in less time

36© 2017 Rogue Wave Software, Inc. All Rights Reserved.

36

Stay tunedConfronting the mission-critical software testing

challengeFeb. 8: Static analysis works for mission-critical systems, why not yours?Compare different techniques for testing by analysis and dive into static code analysis, including the types of problems found, barriers to adoption, and fitting it into various developer environments.

Feb. 22: What if you could eliminate the hidden costs of development?Combat different types of development inefficiency by examining error-prone tasks, waiting for resources, “bug fix crowdsourcing,” and more to learn what the industry is doing about them and what you can do to get ahead.

Page 37: How to achieve security, reliability, and productivity in less time

37© 2017 Rogue Wave Software, Inc. All Rights Reserved.

37