1

GREY BOX TESTINGWeb Apps & Networking

Session 7Boris Grinberg

boris3@gmail.com

2

Session 7 (4 Hours)

• Here are some things that we’ll cover:– What is Software Testing– Worst Software Practices– Heuristics of Software Testing– Testing on Different Platforms and Servers– Web Servers

• Performance, Security and review of specific functionality of most popular Web Servers

– Web Capacity Testing – Load and Stress

What is Software Testing and Why is it Important?

• A brief history of Software engineering and the SDLC.

3

The software industry has evolved through 4 eras, 50’s –60’s, mid 60’s –late 70’s, mid 70’s- mid 80’s, and mid 80’s-present. Each era has its own distinctive characteristics, but over the years the software’s have increased in size and complexity. Several problems are common to almost all of the eras and are discussed below. The Software Crisis dates back to the 1960’s when the primary reasons for this situation were less than acceptable software engineering practices. In the early stages of software there was a lot of interest in computers, a lot of code written but no established standards.

A brief history of Software engineering and the SDLC (continue)Then in early 70’s a lot of computer programs

started failing and people lost confidence and thus an industry crisis was declared. Various reasons leading to the crisis included:

• Hardware advances outpacing the ability to build software for this hardware.

• The ability to build in pace with the demands. • Increasing dependency on software’s• Struggle to build reliable and high quality software• Poor design and inadequate resources.

4

Software Industry Crisis

• This crisis though identified in the early years, exists to date and we have examples of software failures around the world. Software is basically considered a failure if the project is terminated because of costs or overrun schedules, if the project has experienced overruns in excess of 50% of the original or if the software results in client lawsuits. Some examples of failures include failure of Air traffic control systems, failure of medical software, and failure in telecommunication software. The primary reason for these failures other than those mentioned above is due to bad software engineering practices adopted.

5

Worst Software Practices

• Some of the worst software practices include:– No historical software-measurement data.– Rejection of accurate cost estimates.– Failure to use automated estimating and

planning tools.– Excessive, irrational schedule pressure and

creep in user requirements.– Failure to monitor progress and to perform risk

management.– Failure to use design reviews and code

inspections.

6

How to Avoid Worst Software Practices?• To avoid these failures and thus improve the record,

what is needed is a better understanding of the process, better reporting system, better estimation techniques for cost time and quality measures.

• But the question is, what is a process? Process transform inputs to outputs i.e. a product.

A software process is a set of activities, methods and practices involving transformation that people use to develop and maintain software.

7

Software Testing Role

• Having talked about the Software process overall, it is important to identify and relate the role software testing plays not only in producing quality software but also maneuvering the overall process.

• The computer society defines testing as follows: “Testing -- A verification method that applies a controlled set of conditions and cause for the purpose of finding errors. This is the most desirable method of verifying the functional and performance requirements. Test results are documented proof that requirements were met and can be repeated. The resulting data can be reviewed by all concerned for confirmation of capabilities.”

• There may be many definitions of software testing and many which appeal to you from time to time, but its best to start by defining testing and then move on depending on the requirements or needs.

8

Heuristics of Software Testing: Testability

• Software testability is how easily, completely and conveniently a computer program can be tested.

• Software engineers design a computer product, system or program keeping in mind the product testability. Good programmers are willing to do things that will help the testing process and a checklist of possible design points, features and so on can be useful in negotiating with them.

• Here are the two main heuristics of software testing.1. Visibility2. Control

9

Heuristics of Software Testing: Visibility• Visibility is our ability to observe the states and

outputs of the software under test.• Features to improve the visibility are

– Access to Code• Developers must provide full access (source code, infrastructure,

etc) to testers. The Code, change records and design documents should be provided to the testing team. The testing team should read and understand the code.

– Event logging• The events to log include User events, System milestones, Error

handling and completed transactions. The logs may be stored in files, ring buffers in memory, and/or serial ports. Things to be logged include description of event, timestamp, subsystem, resource usage and severity of event. Logging should be adjusted by subsystem and type. Log file report internal errors, help in isolating defects, and give useful information about context, tests, customer usage and test coverage.

10

Heuristics of Software Testing: Visibility• Features to improve the visibility are

– Error detection mechanisms • Data integrity checking and System level error detection (e.g.

Microsoft Appviewer) are useful here. In addition, Assertions and probes with the following features are really helpful:

Code is added to detect internal errors. Assertions abort on error. Probes log errors.

– Resource Monitoring• Memory usage should be monitored to find memory leaks.

States of running methods, threads or processes should be watched (Profiling interfaces may be used for this.). In addition, the configuration values should be dumped. Resource monitoring is of particular concern in applications where the load on the application in real time is estimated to be considerable.

11

Heuristics of Software Testing: Control• Control refers to our ability to provide inputs and reach

states in the software under test.• The features to improve controllability are:• Test Points

– Allow data to be inspected, inserted or modified at points in the software. It is especially useful for dataflow applications. In addition, a pipe and filters architecture provides many opportunities for test points.

• Custom User Interface controls– Custom UI controls often raise serious testability problems with

GUI test drivers. Ensuring testability usually requires:• Adding methods to report necessary information• Customizing test tools to make use of these methods• Getting a tool expert to advise developers on testability and to build the

required support.• Asking third party control vendors regarding support by test tools.

12

Heuristics of Software Testing: Control• The features to improve controllability are:

• Test Interfaces– Interfaces may be provided specifically for testing e.g. Excel– Existing interfaces may be able to support significant testing

e.g. InstallSheild, AutoCAD, Tivoli, etc.

• Fault injection– Error seeding---instrumenting low level I/O code to simulate

errors---makes it much easier to test error handling. It can be handled at both system and application level, Tivoli, etc.

• Installation and setup– Testers should be notified when installation has completed

successfully. They should be able to verify installation, programmatically create sample records and run multiple clients, daemons or servers on a single machine.

13

14

Different Platforms and Servers

• Many problems that current Web sites face have nothing to do with development, but rather with deployment.

• The challenge of building Web sites with reliability, scalability, stability, and manageability needs to be addressed.

• As Web sites begin to handle more business-critical applications, the systems management and operational issues associated with Web development become crucial.

15

Different Platforms and Servers

• A new model for Web development has evolved to address these development and deployment issues. (The Netscape Application Server has popularized Web

application servers. ) The application servers form a clear level of separation between the Web server and data access layers.

• Web sites built using the application server model consist of at least three back-end layers:

• • Web server• • Application server• • Data layers

16

Application Server : Middle Tier

• The logic exists in the middle tier, with application servers handling all data manipulation and HTML page-creation functions.

• The application server approach offers a number of natural advantages, particularly for applications that over time will grow in complexity in terms of business logic or number of users.

17

WebServer: Performance Measurements

• There have been a number of server-side technologies used to increase the power of the server beyond its ability to deliver standard HTML pages; these include CGI scripts, SSL security, ASPs, JSPs, Flash and many more.

• When testing the Web server, there are three important performance measurements:– Response time– Transaction rate– Concurrency

18

The Response Time

• The response time is the total time to send the request to the server and receive the complete response back.

• The response time is closest to the performance that the remote user sees.

• The lower the response time, the better the server.

19

The Transaction Rate

• The transaction rate is the total number of requests that can be processed per second.

• Because modern Web servers are multitasking and/or multithreading, it is possible for a Web server to support a much higher transaction rate than response time.

• The higher the transaction rate, the better the server.

20

Concurrency

• To see how well the server multitasks among multiple simultaneous requests, a concurrency statistic measurement is used. Concurrency is when the average number of simultaneous connections to the server are fired at once.

• Unless the server is overloaded, this number will usually be close to the level specified by the requirements. The higher the number, the better the performance of the server because the system will not overload.

Web Server Testing Features (PERFORMANCE)FEATURE DEFINITION Transactions: The number of times the test script requested the

correct URL.Elapsed time: The number of seconds it took to run the request.Bytes transferred: The total number of bytes sent or received, less HTTP

headers.Response time: The average time it took for the server to respond to

each individual requestTransaction rate: The average number of transactions the server was

able to handle per secondTransferance: The average number of bytes transferred per secondConcurrency: The average number of simultaneous connections the

server was able to handle during the test session.Status code nnn: This indicates how many times a particular HTTP status

code was seen.

21

Web Server Security

• Because Web sites can share information, tight security and encryption have become important issue.

• The most common form of security supported by Web servers is basic authentication, in which users need to provide a user ID and password.

• Most Web servers support such basic authentication, but some servers go a step further and allow access restriction by IP address or host name.

22

Web Server: SSL and Encryption

• Encryption can be used to protect against wire sniffers.

• Web servers use SSL to support encryption.

• All commercial Web servers support SSL, but some support more key-exchange and encryption algorithms.

• SSL creates a secure, encrypted channel between the server and browser by using certificate authentication.

23

What is SSL?

• SSL is an acronym for Secure Sockets Layer, a global standard security technology developed by Netscape in 1994. It creates an encrypted link between a web server and a web browser. The link ensures that all data passed between the web server and browser remains private and secure and is recognized by millions of consumers by a secure padlock which appears in their browser.

24

Web Server Security: Optional Reading

• The SSL Protocol: http://www.lincoln.edu/math/rmyrick/ComputerNetworks/InetReference/ssl-draft/INDEX.HTM• The TLS Protocol. RFC2246:

http://www.ietf.org/rfc/rfc2246.txt• HTTP Over TLS. RFC2818:

http://www.ietf.org/rfc/rfc2818.txt

25

SSL Certificate: Browser Compatibility

• Internet Explorer 5.01 and above• Netscape 4.77 and above• Firefox 0.1 and above• Mozilla 0.6 and above• AOL 5 and above• Opera 8 and above• Safari 1.2 and above

26

Browsers Global Usage StatisticWorldwide %1 Microsoft IE 85.45 %2 Mozilla Firefox 11.51 %3 Apple Safari 1.75 %4 Opera 0.77 %5 Netscape 0.26 %USA %1 Microsoft IE 80.73 %2 Mozilla Firefox 14.07 %3 Apple Safari 3.55 %4 Opera 0.77 %5 Netscape 0.76 %

27

Web Servers List • The most important and popular web

servers:– Apache web server - the HTTP web server– Apache Tomcat– Microsoft Windows Server 2003 Internet

Information Services (IIS)– lighttpd (pronounced "lighty")– Jigsaw– Klone– Sun Java System Web Server– Xitami web server– Zeus web server

28

Apache web server-the HTTP web server

• Free and the most popular web server in the world developed by the Apache Software Foundation.

• Apache web server is an open source software and can be installed and made to work on almost all operating systems including Linux, Unix, Windows, FreeBSD, Mac OS X and more. About 60% of the web server machines run the Apache web server

29

Apache Tomcat

• The Apache Tomcat has been developed to support servlets and JSP scripts. Though it can serve as a standalone server, Tomcat is generally used along with the popular Apache HTTP web server or any other web server. Apache Tomcat is free and open source and can run on different operating systems like Linux, Unix, Windows, Mac OS X, Free BSD.

30

Microsoft Windows Server: Internet Information Services (IIS)

• The IIS (ver 6.0) for Windows Server 2003 operating system has been developed by Microsoft. – It offers higher levels of performance and

security than its predecessor, the Windows 2000 server.

• It is the second most popular server on the web. (Latest version 7.5)

• Read more about Web Servers here.

31

Web Capacity: Load and Stress Testing

• Load and stress testing are critical components of Web testing. This type of testing requires many simultaneous users to make requests during peak activity that will put a large load on the Web server's processor.

• The key to a successful Web site is to have the hardware configured correctly so that it will be powerful enough to meet the demands required.

32

Basic setup for a load testing

33

Load and stress testing are essential to ensure that these demands are met. By performing load testing, you will be able to find performance bottlenecks in your design and setup during the early stages of development.

Figure below illustrates a basic setup for a load testing environment

Load Testing for the Web Test

• The performance of the load or stress test Web site should be monitored with the following in mind: The load test should be able to:

34

• Support all browsers.

• Support all Web servers.

• The tool should be able to simulate at least 1000 users or playback machines.

Load Testing for the Web Test (continue)• The tool should be able to run on Windows, Linux,

Solaris, and most Unix variants.• After the tests are run, you should be able to report

the transactions, URL, and number of users who visited the site.

• The test cases should be assembled in a like fashion to set up test suites.

• There should be a way to:– simulate various users at different connection speeds.– test the different servers and port addresses.– account for the user's cookies.– test for the back-end process, including Active Server Pages,

applets, servlets, plug-ins, ActiveX components, ISAPI, and cgi-bin.

35

Load Testing

• Load testing is a simulation of how a browser will respond to intense use by many individuals. The following are two different types of load tests:

• Single session. A single session should be set up on a browser that will have one or multiple responses. The timing of the data should be put in a file. After the test, you can set up a separate file for report analysis.

• Multiple session. A multiple session should be developed on multiple browsers with one or multiple responses. The multivariate statistical methods may be needed for a complex but general performance model.

36

Memory leaks under stress testing

• Memory leaks are often found under stress testing.

• A memory leak occurs when a test leaves allocated memory behind and does not correctly return the memory to the memory allocation scheme.

• The test seems to run correctly, but after several iterations, available memory is reduced until the system fails.

37

Stress Test Environment (Different Type of the Test Bed) Slide: 1 of 4As you set up your testing environment for

a stress test, you need to make sure you can answer the following questions:• Will my test be able to support all the users

and still maintain performance?• Will my test be able to simulate the number

of transactions that pass through in a matter of hours?

• Will my test be able to uncover whether the system will break?

38

Stress Test Environment. Slide: 2 of 4 • Will my server crash if the load continues

over and over?The test should be set up so that you can

simulate the load; for example:• If you have a remote Web site, you

should be able to monitor up to four Web sites or URLs.

• There should be a way to monitor the load intervals.

• The load test should be able to simulate the SSL (Secure Server).

39

Stress Test Environment. Slide: 3 of 4• The test should be able to simulate

when a user submits the Form Data (GET method).

• The test should be set up to simulate and authenticate the keyword verification.

• The test should be able to simulate up to six email addresses and an alert should occur when there is a failure.

40

Stress Test Environment. Slide: 4 of 4• Remember when stressing your Web

site to give a certain number of users a page to stress test and give them a certain amount of time in which to run the test.

• The key here is to continue to increase the stress level by increasing the number of users until the system performance begins to decrease.

41

Key Features to measure a Stress Test

• Some of the key data features that can help you measure this type of stress test, determine the load, and uncover bottlenecks in the system are:– Amount of memory available and used– The processor time used– The number of requests per second– The amount of time it takes WebPages to be

set up– Server timing errors

42

My Testing Tools recommendations

• There are several types of testing tools that can simulate hundreds of users at server connection speeds.

• I would recommend SilkPerformer from Micro Focus and LoadRunner from Mercury HP.

• On the next 2 slides I will mention a few key features of these tools.

43

SilkPerformer Views

44

SilkPerformer

• Real-world simulations. SilkPerformer accurately emulates the most realistic e-business conditions by simulating a nearly infinite number of simultaneous users and traffic scenarios with a single script. It can also simulate multiple combinations of protocols and computing environments using a single recorder to capture and replay scripts.

• End-to-end reliability. SilkPerformer lets you determine your site's scalability from the earliest stages of development right through final production.

• Firewall support. SilkPerformer maintains firewall integrity while monitoring all application and database servers across any wide area network or Internet infrastructure.

• Agent health control. To ensure valid test results, SilkPerformer continuously monitors the CPU utilization, memory requirements, and responsiveness of each agent.

45

HP LoadRunner Views

46

LoadRunner

• Nonintrusive, real-time performance monitors obtain and display performance data from every tier, server and system component, and diagnostics probes gather code-level data to isolate bottlenecks at the SQL statement or method level. This combination of end user, system-level and code-level visibility dramatically reduces time to problem resolution.

• LoadRunner supports performance testing for a wide range of application environments and protocols, including web, SOA and web services, Ajax, RDP, database, legacy, Citrix, Java, .NET and all major ERP and CRM applications, including PeopleSoft, Oracle, SAP and Siebel. HP LoadRunner has more than 40 non-intrusive monitors tailored for these systems and provides diagnostics for J2EE, .NET, Siebel, Oracle and SAP. HP LoadRunner offers one set of rules for all your enterprise load testing requirements.

• HP LoadRunner integrates with the leading J2EE, Microsoft Visual Studio and Microsoft .NET environments.

47

Web Testing Tools: Selenium vs. iMacros

48

A comparison of features, methods and commands.

Essential features both applications have

• Fully documented commands and specification

• Test scripts can be easily edited • Single-step debugging • Unicode support • Scheduling • Remote Control • Active user forums

49

50

LAB Exercise: iMacros Installation & Use

• Download iMacros Trial (full) version– http://www.iopus.com/download/imacros/– …or from

http://class2.grinberg.biz/sessions.htm

• Install the application• Spend next 20 min to quickly learn at least

1 functionality of the iMacros • Try to create and save one Create• Prepare your questions or concerns

related to this exercise for group discussion

51

Interviews… Boris’s Advice # 7

Google it: Interviewer 1 Interviewer 2

52

Q & A Session

• ? ? ? ? ?• ? ? ? ? ?• ? ? ? ? ?• ? ? ? ? ?• ? ? ? ? ?