GENERAL TESTING PRINCIPLESWhat is Software Testing?

Software testing is an investigation conducted to provide stakeholders with information about the

quality of the product or service under test. Software testing can also provide an objective, independent

view of the software to allow the business to appreciate and understand the risks of software

implementation. Test techniques include the process of executing a program or application with the

intent of finding software bugs (errors or other defects).

It involves the execution of a software component or system component to evaluate one or more

properties of interest. In general, these properties indicate the extent to which the component or

system under test:

meets the requirements that guided its design and development

responds correctly to all kinds of inputs

performs its functions within an acceptable time

is sufficiently usable

can be installed and run in its intended environments

Achieves the general result its stakeholder’s desire.

As the number of possible tests for even simple software components is practically infinite, all software

testing uses some strategy to select tests that are feasible for the available time and resources. As a

result, software testing typically (but not exclusively) attempts to execute a program or application with

the intent of finding software bugs (errors or other defects).

Software testing can provide objective, independent information about the quality of software and risk

of its failure to users and/or sponsors.

Software testing can be conducted as soon as executable software (even if partially complete) exists.

The overall approach to software development often determines when and how testing is conducted.

For example, in a phased process, most testing occurs after system requirements have been defined and

then implemented in testable programs. In contrast, under an Agile approach, requirements,

programming, and testing are often done concurrently.

What does a Software Tester do?

A software tester takes a computer software program through its paces to find glitches or errors that

diminish the program’s usefulness and the user’s satisfaction. In these tests, the main purpose is quality

control, so the tester tries to duplicate common user behaviors along with unusual actions that might

occur infrequently. The tester’s main job is to find weaknesses in the software program that could result

in a malfunction. Software testers may be asked to suggest improvements or modifications that the

original programmers overlooked while creating the code for a software product.

User software has at least two sets of instructions. The software that lets a computer user write letters

balance a budget or create presentations requires a set of instructions, or program. The program users

see — the interface — is different from the program the computer reads to perform the tasks the

software is designed to complete. Each piece of software must also speak to the computer’s operating

system and access resources that it controls.

The codes for each of these instructions combine to create a piece of software. In the course of

developing these software programs, a software tester must verify that all the codes and instructions for

the computer and the user actually work as intended. People who hold these jobs need to understand

how computers work and how people use them.

The proliferation of computer viruses and malware means software testers often run special tests to

check for vulnerabilities in software programs. Computer hackers write or use special programs that

exploit weaknesses in operating system software and user software. Once he discovers these potential

weaknesses, a software tester can communicate with the software designer so the vulnerability to

hackers can be eliminated.

Common skills for software testers include attention to detail and the ability to analyze information.

Logical thinking and the capacity to learn quickly are also important. Programming experience is helpful

because it give the software tester some insight into the probable causes of software errors. A software

tester must also be organized and detail-oriented to document software processes.

This job requires meetings with developers and writing reports along with testing the software. The

majority of software testers have at least one computer certification. Quality engineers are software

testers with specific training and education in the field of software testing. An undergraduate degree

in computer science is generally required to qualify for these jobs. To advance in this career, a graduate

degree is useful.

What does a Software Development Engineer in Test (SDET) do?

SDET stands for Software Development Engineer in Test (or Software Design Engineer in Test). I believe

this title was originated at Microsoft (but have not researched this). It is also used at Amazon.com and

Google.

The SDET role can be compared with that of the SDE (Software Development Engineer) role. The latter is

what most people would think of when they say "Developer". An SDE (depending on level and seniority)

will Architect, Design, and Implement software systems that will be used by the target end users. SDEs

are responsible for the quality of their software and will implement testing (often unit tests), as well as

seek reviews for both their designs and their code. SDEs will often have influence on the software

processes employed, and be responsible for following best practices in software development.

The SDET is also a developer. The SDET must have knowledge of what represents good software design.

The SDET must be able to create high quality, maintainable, and performant code. The codes generally

created by the SDET however are for automated test cases and the frameworks to execute and report

them. An SDET's knowledge of software design is often focused on testability, robustness, and

performance, and they usually play a contributory or reviewer role in the creation of designs for

production software. A SDETs skill set will often include more experience in software processes and how

to test software. Testing software is generally done so as to assess and mitigate risk, and SDETs need to

be expert in this. So an SDET can be summarized as having development skills and software quality skills.

While SDEs should have this also, SDEs balance more towards the former while SDETs balance more

towards the latter.

What is an SDE-T?

Programming experience a must

o Key here is that the candidate is a Developer. Strong QA candidates without

development experience are generally not acceptable for this role.

Test Automation

o Writing code to test code: UI and API level testing

o Creating, modifying, or designing test frameworks

o Good: Selenium, JUnit, NUnit, TestNG, etc.

o Using “canned” products like QuickTest Pro, TestDirector, SilkTest is generally *not*

sufficient

SDLC and Software Processes

o Agile/Scrum a plus

QA interest / enthusiasm / experience

o Created Test Cases, Test Plans

o Debug / troubleshoot issues that they found (deep dive)

o Bug reporting / triage / prioritization

o Was responsible for guiding quality on production release

o Functional, load, stress, user interaction testing

Customer focus

o Customer facing experience a plus

"Soft" Skills

o Communication

o Driving consensus

o Leadership

o Creativity

Technologies

o Must have experience programming a high-level OO language (C#, Java, C++, or C#)

C#/.NET experience highly preferred

o RDBMS, SQL, stored procedures. SQL Server preferred.

o Web technologies: HTML, JavaScript, web services, XML, CSS

How to sell the SDE-T role to a developer

Breadth of experience over narrow focus

End to end product expertise. From development to user interaction

Subject Matter Expert on product. Has input on product direction.

Career “big fish in a small pond”… a strong contributor can really make a name for him or

herself. Easier than in Dev because do not get “pigeon holed” and because the QA field is more

exclusive.

What’s the difference between a Tester and an SDET?

The Microsoft career site has two excellent definitions for the Software Test Engineer and Software

Design Engineer in Test roles at Microsoft. These clearly defined role definitions have very vague

applications at Microsoft. After all, aren’t we all just assuring that the software that we ship is the

highest quality possible?

Historically, it seems to have made a lot of sense that these two positions had distinct roles. Years ago,

Software Test Engineers were testers who had limited programming experience. Software Test

Engineers have been used as monkey labor- “Run this test fifty times, and call me if this breakpoint gets

hit”. STEs didn’t need a strong CS background to get their jobs done- they just needed a lot of patience,

and basic understanding of using computers. The Software Design Engineer in Test role was reserved for

Testers who were capable of writing tools and/or automation. Some STE hires have written tools in their

first half year on the job having their job title switched by management to SDE/T.

It’s easy to understand the difference between STE and SDET with a black and white example. If you’re

testing an API, then yes, you’ll surely be an SDET (how can you do any testing without writing a line of

code?) Conversely, if you exclusively black box test a user interface (without tools/automation), then

you’d have the job title of STE. The gray area between these two extreme examples is huge, though.

Most, if not all, STEs end up writing at least some code. I'm sure there are plenty of STEs that in practice

write more code than SDETs! This difference is particularly pronounced across different product teams

at Microsoft.

Management seems to be pushing for a unification of these two roles, probably for this reason. Both

roles dedicate themselves to product quality; both are reliant on strong fundamental understanding and

application of CS principles. A quick look on the career website seems to back this up—there are

currently more active postings for SDETs, compared with those for the job title of STE. This doesn’t mean

software testing is going away—it’s rather a reflection of the increasingly technical nature of testing

software at Microsoft. It looks like the job title that the company is shifting to is SDET, which is more a

mouthful but also more impressive sounding.

What’s the Software Development Lifecycle (SDLC)?SDLC is a process followed for a software project, within a software organization. It consists of a detailed plan describing how to develop, maintain, replace and alter or enhance specific software. The life cycle defines a methodology for improving the quality of software and the overall development process.

The following figure is a graphical representation of the various stages of a typical SDLC.

A typical Software Development life cycle consists of the following stages:

Stage 1: Planning and Requirement Analysis

In this phase of SDLC, suitable necessities of system are accumulated. All adjacent methods should be in focus. All types of estimation and examination of user needs are done in this phase.

Requirement analysis is the most important and fundamental stage in SDLC. It is performed by the senior members of the team with inputs from the customer, the sales department, market surveys and domain experts in the industry. This information is then used to plan the basic project approach and to conduct product feasibility study in the economical, operational, and technical areas.

Planning for the quality assurance requirements and identification of the risks associated with the project is also done in the planning stage. The outcome of the technical feasibility study is to define the various technical approaches that can be followed to implement the project successfully with minimum risks.

Stage 2: Defining Requirements

Once the requirement analysis is done the next step is to clearly define and document the product requirements and get them approved from the customer or the market analysts. This is done through SRS (Software Requirement Specification) document which consists of all the product requirements to be designed and developed during the project life cycle.

Stage 3: Designing the Product Architecture

In the next phase implementation of project is done. Respect to the system design, correct development is made to expand that design. According to the project programming language will be chosen.

SRS is the reference for product architects to come out with the best architecture for the product to be developed. Based on the requirements specified in SRS, usually more than one design approach for the product architecture is proposed and documented in a DDS—Design Document Specification.

This DDS is reviewed by all the important stakeholders and based on various parameters as risk assessment, product robustness, design modularity, and budget and time constraints. The best design approach is selected for the product.

A design approach clearly defines all the architectural modules of the product along with its communication and data flow representation with the external and third party modules (if any). The internal design of all the modules of the proposed architecture should be clearly defined with the minutest of the details in DDS.

Stage 4: Building or Developing the Product

In this stage of SDLC the actual development starts and the product is built. The programming code is generated as per DDS during this stage. If the design is performed in a detailed and organized manner, code generation can be accomplished without much hassle.

Developers have to follow the coding guidelines defined by their organization and programming tools like compilers, interpreters, debuggers etc. are used to generate the code. Different high level programming languages such as C, C++, Pascal, Java, and PHP are used for coding. The programming language is chosen with respect to the type of software being developed.

Stage 5: Testing the Product

After the implementation phase, system testing phase take place to recognize the result of application. Testing is done to recognize the original result and the predictable result.

This stage is usually a subset of all the stages as in the modern SDLC models, the testing activities are mostly involved in all the stages of SDLC. However this stage refers to the testing only stage of the

product where products defects are reported, tracked, fixed and retested, until the product reaches the quality standards defined in the SRS.

Stage 6: Deployment in the Market and Maintenance

It is the ultimate phase of SDLC, where the application which is implemented is spread to users who are answerable for conserving and using it for appropriate actions. The implemented application should be available for any adjustment to do in coding.

Once the product is tested and ready to be deployed it is released formally in the appropriate market. Sometime product deployment happens in stages as per the organizations business strategy. The product may first be released in a limited segment and tested in the real business environment (UAT- User acceptance testing).

Then based on the feedback, the product may be released as it is or with suggested enhancements in the targeting market segment. After the product is released in the market, its maintenance is done for the existing customer base.

SDLC ModelsThere are various software development life cycle models defined and designed which are followed during software development process. These models are also referred as "Software Development Process Models". Each process model follows a Series of steps unique to its type, in order to ensure success in process of software development.

Following are the most important and popular SDLC models followed in the industry:

Waterfall Model

The waterfall model is a sequential design process, used in software development processes, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, production/implementation and maintenance.

The waterfall development model originates in the manufacturing and construction industries: highly structured physical environments in which after-the-fact changes are prohibitively costly, if not impossible. Since no formal software development methodologies existed at the time, this hardware-oriented model was simply adapted for software development.

Iterative Model

Iterative and Incremental development is any combination of both iterative design or iterative method and incremental build model for software development. The combination is of long standing and has been widely suggested for large development efforts. For example, the 1985 DOD-STD-2167 mentions: "During software development, more than just one iteration of the software development cycle may be in progress at the same time." and "This process may be described as an 'evolutionary acquisition' or 'incremental build' approach." The relationship between iterations and increments is determined by the overall software development methodology and software development process. The exact number and nature of the particular incremental builds and what is iterated will be specific to each individual development effort.

Iterative and incremental developments are essential parts of the Modified waterfall models, Rational Unified Process, Extreme Programming and generally the various agile software development frameworks. It follows a similar process to the plan-do-check-act cycle of business process improvement.

Spiral Model

The spiral model is a risk-driven process model generator for software projects. Based on the unique risk patterns of a given project, the spiral model guides a team to adopt elements of one or more process models, such as incremental, waterfall, or evolutionary prototyping.

V-Model

The V-model represents a software development process (also applicable to hardware development) which may be considered an extension of the waterfall model. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape. The V-Model demonstrates the relationships between each phase of the development life cycle and its associated phase of testing. The horizontal and vertical axes represents time or project completeness (left-to-right) and level of abstraction (coarsest-grain abstraction uppermost), respectively.

Big Bang Model

The Big Bang Model is just like the cosmological model that it is named after namely one in which we put huge amounts of matter (people or money) and these are put together, a lot of energy is expended, often violently, and the outcome is either the perfect software product or not.

The beauty of this model is that it is simple. There is little planning, scheduling, or formal development process. All the effort is spent developing the software and writing the code. This model is usually used in small projects, since the developer does the requirement analysis, writes code and develops software.

The Others

Other related methodologies are Agile Model, RAD Model, Rapid Application Development and Prototyping Models.

What’s the role of the Tester in the SDLC?In every company, testing is the most vital and precious stage in the software development life cycle. But, the technique by which it is performed, dissimilar from one company to another company. Software testing has turned into a part of programming and it is good to begin testing from the first stage, to avoid complexity by repairing the errors at the last stage. Also in software development life cycle (SDLC) testing includes pick up consistency, interpretation and other major elements, which may explain under software requirement specification. Clients may stay further for software delivery, but they don't akin to work with imperfect software. It is desirable to perform the testing process from the first stages, with considerations made to the Software Development Life Cycle to elude any difficulty.

Extent of Software Testing

The basic job of software testing is to identify errors in order to reveal and spot it. The extent of software testing consists of implementation of that code in different domains and also to look at what features of the code does the software perform, what it should be doing, and methods with respect to the condition. It is proposed to begin testing from the first phase of the software development. This not only aids in correcting the faults earlier to the last step, but also decreases the reworking of getting errors in the first step every time. It saves time as well as cost. It is a continuous method, which can be nonstop, but does stop for the need of time and resources. The basic need of the testing is to provide the best quality product while using minimal time and resources. The test engineer has to pursue some technical way by which they can review that all the necessary points for testing have been covered. A register should be created for keeping records of the day to day test cases. Presently in the IT Field, a testing group might be divided from the development team. There are several roles for testing team members.

Estimating Software Testing

There is a requirement for estimating the cost of the software, at the phase of implementation, as well as after the software is completed for delivery. Though it is complicated to estimate the conceptual controls, the controls are also very important for the estimation. The components that cannot be estimated have to be restricted. There are some vital uses of estimating the system.

Software estimation is used to limit problems with:

Exceeding Costs

Identifying the problems

Illuminating objectives

Evaluation of the system

Standards of the code develop

Upgrading under implemented code

It helps to estimate the value of the application, price and attempt evaluation, set of information, output and achievement assessment.

Role of Testing in SDLC

Inception Phase

o In this phase, a test engineer will get an opportunity to identify the necessities of the project. Normally the data is recorded by the architecture team in the architectural reference document. Data design, information design, system design are the main issues in this phase.

Elaboration Phase

o In this phase, a test engineer will get an opportunity to identify how the project is planned. This is a major phase, where the entire design of the project is documented in the JAD phase in the System requirement document, business requirement document, product requirement document commercial use cases. Planner, Commercial reviewer, project organization, execution, testing, maintenance of project teams etc. are part of the JAD phase to sign-off on the completed document.

Construction Phase

o In this phase, programmers play an important role of building the application. Here the tester group follows the programming group to identify several adjustments taken by the system. Any fault overlooked by programmers, a tester can always raise the issue to the programmers to solve. A testing group requires developing the high level scenarios (HLS) on basis of the elaboration phase. High level scenarios may have more than one test case. A tester should ensure that all the necessities are discovered to a test case by a quality assurance standard. It is mandatory to record test cases on the basis of all probable references of the newest modernized data and also signed-off.

Transition Phase

o In this phase, if any fault or errors arise, they are tested again and it goes though the regression testing. With the help of regression testing, consistent systems develop. With these testing methods, any fundamental result can be converted into a tough and consistent system.

Software Development Life Cycle Model

There are different types of Software Development Life Cycle models and every model is used in testing stage. It makes testing a very essential element in any SDLC. With the help of different types of testing (e.g. assimilation tests, element testing, user recognition testing system testing and regression tests etc.) coders may produce a consistent and reliable application. Testing also includes: test analysis, test plan, test design and test execution.

What is Quality? In the context of software engineering, software quality refers to two related but distinct notions that exist wherever quality is defined in a business context:

Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specifications. That attribute can also be described as the fitness for purpose of a piece of software or how it compares to competitors in the marketplace as a worthwhile product;

Software structural quality refers to how it meets non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability, the degree to which the software was produced correctly.

Structural quality is evaluated through the analysis of the software inner structure, its source code, at the unit level, the technology level and the system level, which is in effect how its architecture adheres to sound principles of software architecture outlined in a paper on the topic by OMG. In contrast, functional quality is typically enforced and measured through software testing.

Historically, the structure, classification and terminology of attributes and metrics applicable to software quality management have been derived or extracted from the ISO 9126-3 and the subsequent ISO 25000:2005 quality model, also known as SQuaRE. Based on these models, the Consortium for IT Software Quality (CISQ) has defined five major desirable structural characteristics needed for a piece of software to provide business value: Reliability, Efficiency, Security, Maintainability and (adequate) Size.

Software quality measurement quantifies to what extent a software or system rates along each of these five dimensions. An aggregated measure of software quality can be computed through a qualitative or a quantitative scoring scheme or a mix of both and then a weighting system reflecting the priorities. This view of software quality being positioned on a linear continuum is supplemented by the analysis of "critical programming errors" that under specific circumstances can lead to catastrophic outages or performance degradations that make a given system unsuitable for use regardless of rating based on aggregated measurements. Such programming errors found at the system level represent up to 90% of production issues, whilst at the unit-level, even if far more numerous, programming errors account for less than 10% of production issues. As a consequence, code quality without the context of the whole system, as W. Edwards Deming described it, has limited value.

To view, explore, analyze, and communicate software quality measurements, concepts and techniques of information visualization provide visual, interactive means useful, in particular, if several software quality measures have to be related to each other or to components of a software or system. For example, software maps represent a specialized approach that "can express and combine information about software development, software quality, and system dynamics".

Motivation

Measuring software quality is motivated by at least two reasons:

Risk Management: Software failure has caused more than inconvenience. Software errors have caused human fatalities. The causes have ranged from poorly designed user interfaces to direct programming errors. An example of a programming error that led to multiple deaths is discussed in Dr. Leveson's paper. This resulted in requirements for the development of some types of software, particularly and historically for software embedded in medical and other devices that regulate critical infrastructures: "[Engineers who write embedded software] see Java programs stalling for one third of a second to perform garbage collection and update the user interface, and they envision airplanes falling out of the sky.” In the United States, within the Federal Aviation Administration (FAA), the FAA Aircraft Certification Service provides software programs, policy, guidance and training, focus on software and Complex Electronic Hardware that has an effect on the airborne product (a "product" is an aircraft, an engine, or a propeller).

Cost Management: As in any other fields of engineering, an application with good structural software quality costs less to maintain and is easier to understand and change in response to pressing business needs. Industry data demonstrate that poor application structural quality in core business applications (such as Enterprise, Customer Relationship Management (CRM) or large transaction processing systems in financial services) results in cost and schedule overruns and creates waste in the form of rework (up to 45% of development time in some organizations ). Moreover, poor structural quality is strongly correlated with high-impact business disruptions due to corrupted data, application outages, security breaches, and performance problems.

However, the distinction between measuring and improving software quality in an embedded system (with emphasis on risk management) and software quality in business software (with emphasis on cost and maintainability management) is becoming somewhat irrelevant. Embedded systems now often include a user interface and their designers are as much concerned with issues affecting usability and user productivity as their counterparts who focus on business applications. The latter are in turn looking at ERP or CRM system as a corporate nervous system whose uptime and performance are vital to the well-being of the enterprise. This convergence is most visible in mobile computing: a user who accesses an ERP application on their smartphone is depending on the quality of software across all types of software layers.

Both types of software now use multi-layered technology stacks and complex architecture so software quality analysis and measurement have to be managed in a comprehensive and consistent manner, decoupled from the software's ultimate purpose or use. In both cases, engineers and management need to be able to make rational decisions based on measurement and fact-based analysis in adherence to the precept "In God (we) trust. All others bring data".

Definitions

There are many different definitions of quality. For some it is the "capability of a software product to conform to requirements." (ISO/IEC 9001) while for others it can be synonymous with "customer value" or even defect level.

The first definition of quality History remembers is from Shewhart in the beginning of 20th century: There are two common aspects of quality: one of them has to do with the consideration of the quality of a thing as an objective reality independent of the existence of man. The other has to do with what we think, feel or sense as a result of the objective reality. In other words, there is a subjective side of quality.

Kitchenham, Pfleeger, and Garvin's five perspectives on quality

Kitchenham and Pfleeger, further reporting the teachings of David Garvin, identify five different perspectives on quality:

The transcendental perspective deals with the metaphysical aspect of quality. In this view of quality, it is "something toward which we strive as an ideal, but may never implement completely". It can hardly be defined, but is similar to what a federal judge once commented about obscenity: "I know it when I see it".

The user perspective is concerned with the appropriateness of the product for a given context of use. Whereas the transcendental view is ethereal, the user view is more concrete, grounded in the product characteristics that meet user’s needs.

The manufacturing perspective represents quality as conformance to requirements. This aspect of quality is stressed by standards such as ISO 9001, which defines quality as "the degree to which a set of inherent characteristics fulfills requirements" (ISO/IEC 9001).

The product perspective implies that quality can be appreciated by measuring the inherent characteristics of the product.

The final perspective of quality is value-based. This perspective recognizes that the different perspectives of quality may have different importance, or value, to various stakeholders.

Software quality according to Deming

"The problem inherent in attempts to define the quality of a product, almost any product, was stated by the master Walter A. Shewhart. The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay. This is not easy, and as soon as one feels fairly successful in the endeavor, he finds that the needs of the consumer have changed, competitors have moved in, etc."

Software quality according to Feigenbaum

"Quality is a customer determination, not an engineer's determination, not a marketing determination, nor a general management determination. It is based on the customer's actual experience with the product or service, measured against his or her requirements -- stated or unstated, conscious or merely sensed, technically operational or entirely subjective -- and always representing a moving target in a competitive market".

Software quality according to Juran

"The word quality has multiple meanings. Two of these meanings dominate the use of the word: 1. Quality consists of those product features which meet the need of customers and thereby provide product satisfaction. 2. Quality consists of freedom from deficiencies. Nevertheless, in a handbook such as this it is convenient to standardize on a short definition of the word quality as "fitness for use"."

CISQ's quality model

Even though "quality is a perceptual, conditional and somewhat subjective attribute and may be understood differently by different people" (as noted in the article on quality), software structural quality characteristics have been clearly defined by the Consortium for IT Software Quality (CISQ). Under the guidance of Bill Curtis, co-author of the Capability Maturity Model framework and CISQ's first Director; and Capers Jones, CISQ's Distinguished Advisor, CISQ has defined five major desirable characteristics of a piece of software needed to provide business value. In the House of Quality model, these are "What’s" that need to be achieved:

Reliability

o An attribute of resiliency and structural solidity—Reliability measures the level of risk and the likelihood of potential application failures. It also measures the defects injected due to modifications made to the software (its “stability” as termed by ISO). The goal for checking and monitoring Reliability is to reduce and prevent application downtime, application outages and errors that directly affect users, and enhance the image of IT and its impact on a company’s business performance.

Efficiency

o The source code and software architecture attributes are the elements that ensure high performance once the application is in run-time mode. Efficiency is especially important for applications in high execution speed environments such as algorithmic or transactional processing where performance and scalability are paramount. An analysis of source code efficiency and scalability provides a clear picture of the latent business risks and the harm they can cause to customer satisfaction due to response-time degradation.

Security

o A measure of the likelihood of potential security breaches due to poor coding practices and architecture. This quantifies the risk of encountering critical vulnerabilities that damage the business.

Maintainability

o Maintainability includes the notion of adaptability, portability and transferability (from one development team to another). Measuring and monitoring maintainability is a must for mission-critical applications where change is driven by tight time-to-market schedules and where it is important for IT to remain responsive to business-driven changes. It is also essential to keep maintenance costs under control.

Size

o While not a quality attribute per se, the sizing of source code is a software characteristic that obviously impacts maintainability. Combined with the above quality characteristics, software size can be used to assess the amount of work produced and to be done by teams, as well as their productivity through correlation with time-sheet data, and other SDLC-related metrics.

Software functional quality is defined as conformance to explicitly stated functional requirements, identified for example using Voice of the Customer analysis (part of the Design for Six Sigma toolkit and/or documented through use cases) and the level of satisfaction experienced by end-users. The latter is referred as to as usability and is concerned with how intuitive and responsive the user interface is, how easily simple and complex operations can be performed, and how useful error messages are. Typically, software testing practices and tools ensure that a piece of software behaves in compliance with the original design, planned user experience and desired testability, i.e. a piece of software's disposition to support acceptance criteria.

The dual structural/functional dimension of software quality is consistent with the model proposed in Steve McConnell's Code Complete which divides software characteristics into two pieces: internal and external quality characteristics. External quality characteristics are those parts of a product that face its users, where internal quality characteristics are those that do not.

Alternative approaches

One of the challenges in defining quality is that "everyone feels they understand it" and other definitions of software quality could be based on extending the various descriptions of the concept of quality used in business.

Dr. Tom DeMarco has proposed that "a product's quality is a function of how much it changes the world for the better." This can be interpreted as meaning that functional quality and user satisfaction are more important than structural quality in determining software quality.

Another definition, coined by Gerald Weinberg in Quality Software Management: Systems Thinking is "Quality is value to some person." this definition stresses that quality is inherently subjective—different people will experience the quality of the same software differently. The strength of this definition is the questions it invites software teams to consider, such as "Who are the people we want to value our software?" and "What will be valuable to them?"

What’s the business need for Software Testers?There is a lot more to being a software tester than just understanding how to detect and fix bugs. Quality assurance professionals work to ensure that the apps they help create live up to user and stakeholder standards throughout its lifecycle. This, of course, is often easier said than done, as the overall view of a specific project can often be lost with all of the other tasks on a QA employee's plate. For this reason, it's critical for testers to be business savvy and have the organization's strategy in mind at all times.

Building up business savvy

Whereas traditional development methods often siloed off testers from the company side, agile environments make collaboration necessary across all fronts. According to PM College, there are a few habits that business savvy testers should adhere to, such as managing projects for organization impact, communicating to drive results and demonstrating project value to potential stakeholders. These practices can only be achieved if a tester fully understands what the business's goals are and how a product fits into the big picture.

QA professionals should be able to detail how an app can help the organization further its development, improve processes and streamline employee workflows. If a tester cannot do this, they will likely not be able to produce software that has the type of functionality and features that users are expecting. Even a basic understanding of the company's strategy could mean the difference between a successful program and one that misses the mark.

"A business-savvy project manager can quickly summarize the business reasons for engaging in the project, including how it is aligned with the business goals of the company," PM College stated. "The business-savvy project manager moves away from concentrating on the technical details of the project and focuses on the overall value of the project – and of project management in general."

Take an overarching perspective

Although QA is only one part of a business, these employees must be able to look at the big picture. Tech Product Management contributor Daniel Elizalde noted that knowing the industry, how the product contributes to the company's bottom line and the complete product lifecycle from the organization's perspective will all be critical factors. Understanding these elements will help QA teams create a functional app from the very beginning.

Testers might also be able to come up with specific tests that will better detect bugs that could affect the program's performance. Elizalde noted that having business savvy will help QA teams define how users will be trained and how to incorporate feedback for further development. These aspects will be necessary to establish before launch, ensuring that the integration goes smoothly and that all needs are met on a rolling basis.

"A product usually starts out of a business need," Elizalde wrote. "From there, it requires market and user research to validate the idea and then funding for the development effort. Once the actual product

development starts, there are the typical stages of UX, development, QA, etc., but in parallel, there are other tracks going on."

Testing correlates to company development

The business and its strategy directly affect how much testing is required for various projects and the approaches QA teams take toward software development. Industry experts Michael Kelly and Jeanette Thebeau have noted that an organization's stage impacts how much it values testing. For example, many decision-makers don't mind a few bugs early on in the company's lifecycle, since the main goal to push the product to beta. Following in the footsteps of giants like Google, which has had products in beta for years; this approach seems to make sense for businesses that are just starting out.

On the flip side, if an organization is growing, testing becomes paramount. Specifically when the testing is tied to a major product launch, businesses should take extra precautions to ensure that they have the skills on hand to meet their goals and handle any unforeseen challenges.

An organization's business strategy is a critical aspect for testers to keep in mind as they go through their projects. By having a better overview of the company's goals and development, QA teams will be able to better evaluate their software and ensure that it meets stakeholder expectations.

DIFFERENT TYPES OF SOFTWARE TESTINGFunctional testing

Functional testing refers to activities that verify a specific action or function of the code. These are usually found in the code requirements documentation, although some development methodologies work from use cases or user stories. Functional tests tend to answer the question of "can the user do this" or "does this particular feature work."

Usability testing

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system. This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users.

Usability testing focuses on measuring a human-made product's capacity to meet its intended purpose. Examples of products that commonly benefit from usability testing are foods, consumer products, web sites or web applications, computer interfaces, documents, and devices. Usability testing measures the usability, or ease of use, of a specific object or set of objects, whereas general human-computer interaction studies attempt to formulate universal principles.

Acceptance testing

Acceptance testing can mean one of two things:

A smoke test is used as an acceptance test prior to introducing a new build to the main testing process, i.e. before integration or regression.

Acceptance testing performed by the customer, often in their lab environment on their own hardware, is known as user acceptance testing (UAT). Acceptance testing may be performed as part of the hand-off process between any two phases of development.

Manual testing

Manual testing is the process of manually testing software for defects. It requires a tester to play the role of an end user and use most of all features of the application to ensure correct behavior. To ensure completeness of testing, the tester often follows a written test plan that leads them through a set of important test cases.

Automated testing

Many programming groups are relying more and more on automated testing, especially groups that use test-driven development. There are many frameworks to write tests in, and continuous integration software will run tests automatically every time code is checked into a version control system.

While automation cannot reproduce everything that a human can do (and all the ways they think of doing it), it can be very useful for regression testing. However, it does require a well-developed test suite of testing scripts in order to be truly useful.

Unit testing

Unit testing, also known as component testing refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors.

These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner cases or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to ensure that the building blocks of the software work independently from each other.

Unit testing is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs. It is performed by the software developer or engineer during the construction phase of the software development lifecycle. Rather than replace traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors before code is promoted to QA; this strategy is intended to increase the quality of the resulting software as well as the efficiency of the overall development and QA process.

Depending on the organization's expectations for software development, unit testing might include static code analysis, data flow analysis, metrics analysis, peer code reviews, code coverage analysis and other software verification practices.

Integration testing

Integration testing is any type of software testing that seeks to verify the interfaces between components against a software design. Software components may be integrated in an iterative way or all together ("big bang"). Normally the former is considered a better practice since it allows interface issues to be located more quickly and fixed.

Integration testing works to expose defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a system.

Accessibility testing

Accessibility testing may include compliance with standards such as:

Americans with Disabilities Act of 1990

Section 508 Amendment to the Rehabilitation Act of 1973

Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C)

Alpha - Beta testing

Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.

Beta testing comes after alpha testing and can be considered a form of external user acceptance testing. Versions of the software, known as beta versions, are released to a limited audience outside of the programming team known as beta testers. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Beta versions can be made available to the open public to increase the feedback field to a maximal number of future users and to deliver value earlier, for an extended or even infinite period of time (perpetual beta).

Smoke testing/Sanity testing

Sanity testing determines whether it is reasonable to proceed with further testing.

Smoke testing consists of minimal attempts to operate the software, designed to determine whether there are any basic problems that will prevent it from working at all. Such tests can be used as build verification test.

Ad-hoc testing

Ad hoc testing is a commonly used term for software testing performed without planning and documentation, but can be applied to early scientific experimental studies.

The tests are intended to be run only once, unless a defect is discovered. Ad hoc testing is the least formal test method. As such, it has been criticized because it is not structured and hence defects found using this method may be harder to reproduce (since there are no written test cases). However, the strength of ad hoc testing is that important defects can be found quickly.

It is performed by improvisation: the tester seeks to find bugs by any means that seem appropriate. Ad hoc testing can be seen as a light version of error guessing, which itself is a light version of exploratory testing.

Regression testing

Regression testing focuses on finding defects after a major code change has occurred. Specifically, it seeks to uncover software regressions, as degraded or lost features, including old bugs that have come back. Such regressions occur whenever software functionality that was previously working, correctly, stops working as intended. Typically, regressions occur as an unintended consequence of program changes, when the newly developed part of the software collides with the previously existing code. Common methods of regression testing include re-running previous sets of test-cases and checking whether previously fixed faults have re-emerged. The depth of testing depends on the phase in the release process and the risk of the added features. They can either be complete, for changes added late

in the release or deemed to be risky, or be very shallow, consisting of positive tests on each feature, if the changes are early in the release or deemed to be of low risk. Regression testing is typically the largest test effort in commercial software development, due to checking numerous details in prior software features, and even new software can be developed while using some old test-cases to test parts of the new design to ensure prior functionality is still supported.

Compatibility testing

A common cause of software failure (real or perceived) is a lack of its compatibility with other application software, operating systems (or operating system versions, old or new), or target environments that differ greatly from the original (such as a terminal or GUI application intended to be run on the desktop now being required to become a web application, which must render in a web browser). For example, in the case of a lack of backward compatibility, this can occur because the programmers develop and test software only on the latest version of the target environment, which not all users may be running. This results in the unintended consequence that the latest work may not function on earlier versions of the target environment or on older hardware that earlier versions of the target environment was capable of using. Sometimes such issues can be fixed by proactively abstracting operating system functionality into a separate program module or library.

Performance testing

Performance testing is generally executed to determine how a system or sub-system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate, or verify other quality attributes of the system, such as scalability, reliability and resource usage.

There is little agreement on what the specific goals of performance testing are. The terms load testing, performance testing, scalability testing, and volume testing, are often used interchangeably.

Real-time software systems have strict timing constraints. To test if timing constraints are met, real-time testing is used.

Load testing

Load testing is primarily concerned with testing that the system can continue to operate under a specific load, whether that is large quantities of data or a large number of users. This is generally referred to as software scalability. The related load testing activity of when performed as a non-functional activity is often referred to as endurance testing. Volume testing is a way to test software functions even when certain components (for example a file or database) increase radically in size. Stress testing is a way to test reliability under unexpected or rare workloads. Stability testing (often referred to as load or endurance testing) checks to see if the software can continuously function well in or above an acceptable period.

A/B Testing

A/B testing is basically a comparison of two outputs, generally when only one variable has changed: run a test, change one thing, run the test again, compare the results. This is more useful with more small-

scale situations, but very useful in fine-tuning any program. With more complex projects, multivariate testing can be done.

Black-box testing

Black-box testing treats the software as a "black box", examining functionality without any knowledge of internal implementation. The testers are only aware of what the software is supposed to do, not how it does it. Black-box testing methods include: equivalence partitioning, boundary value analysis, all-pairs testing, state transition tables, decision table testing, fuzz testing, model-based testing, use case testing, exploratory testing and specification-based testing.

White-box testing

White-box testing (also known as clear box testing, glass box testing, and transparent box testing and structural testing) tests internal structures or workings of a program, as opposed to the functionality exposed to the end-user. In white-box testing an internal perspective of the system, as well as programming skills, are used to design test cases. The tester chooses inputs to exercise paths through the code and determine the appropriate outputs. This is analogous to testing nodes in a circuit, e.g. in-circuit testing (ICT).

While white-box testing can be applied at the unit, integration and system levels of the software testing process, it is usually done at the unit level. It can test paths within a unit, paths between units during integration, and between subsystems during a system–level test. Though this method of test design can uncover many errors or problems, it might not detect unimplemented parts of the specification or missing requirements.

Techniques used in white-box testing include:

API testing – testing of the application using public and private APIs (application programming interfaces)

Code coverage – creating tests to satisfy some criteria of code coverage (e.g., the test designer can create tests to cause all statements in the program to be executed at least once)

Fault injection methods – intentionally introducing faults to gauge the efficacy of testing strategies

Mutation testing methods

Static testing methods

Code coverage tools can evaluate the completeness of a test suite that was created with any method, including black-box testing. This allows the software team to examine parts of a system that are rarely tested and ensures that the most important function points have been tested. Code coverage as a software metric can be reported as a percentage for:

Function coverage, which reports on functions executed

Statement coverage, which reports on the number of lines executed to complete the test

Decision coverage, which reports on whether both the True and the False branch of a given test has been executed

100% statement coverage ensures that all code paths or branches (in terms of control flow) are executed at least once. This is helpful in ensuring correct functionality, but not sufficient since the same code may process different inputs correctly or incorrectly.

TESTING DOCUMENTATIONSWhat’s Requirements Document?

A requirements document explains why a product is needed, puts the product in context, and describes what the finished product will be like. A large part of the requirements document is the formal list of requirements.

What Are Requirements?

Requirements include descriptions of system properties, specifications for how the system should work, and constraints placed upon the development process. Generally, requirements are statements of what a system should do rather than how it should do it. The answers to how questions fall into the realm of design. Requirements specifications should not include design solutions (except for interface requirements, which often include embedded design).

Requirements come from end users, from customers, and sometimes from developers. End users tend to state requirements in descriptive or narrative terms ("I'd like a welcome screen with links to the things I use regularly, and I want to be able to select from a couple of different color schemes for the welcome screen") which might need to be broken down into individual requirement statements.

Customers, who may well be different from end users, are the people who are paying for the development of the system. Their requirements will often be stated in terms of costs or scheduling issues. Developers might have requirements related to system performance and other technical topics.It's important to have all these groups contribute to the requirements document to create a fuller description of the system. The practice of including these groups also helps to ensure that everyone is in agreement about what is to be done before development begins.

Requirements documents usually include user, system, and interface requirements; other classes of requirements are included as needed. User requirements are written from the point of view of end users, and are generally expressed in narrative form: The user must be able to change the color scheme of the welcome screen. System requirements are detailed specifications describing the functions the system needs to do. These are usually more technical in nature: The system will include four preset color schemes for the welcome screen. Colors must be specified for the page background, the text, visited links, unvisited links, active links, and buttons (base, highlight, and shadow). Interface requirements specify how the interface (the part of the system that users see and interact with) will look and behave. Interface requirements are often expressed as screen mock-ups; narratives or lists are also used.

What Else Goes Into a Requirements Document?

Besides listing requirements, the document should explain why the product is needed, put the product in context for development, and describe what the finished product will be like. To put the product in context for development, describe the development tools that will be used, the project budget and staffing situations, and the approximate development schedule. Include details about how the finished product will be distributed or accessed, and any other pertinent information that affects development.

Writing a Requirements Document

The document should include an overview of the finished product in narrative and/or graphical mockup form for those who want to get a grasp of the project without reading all the requirements. An overview is also useful for people who will be reading the requirements, so that they can have a general understanding of the product before delving into the specifications. Obviously, the scale of the project will influence the contents and length of a requirements document. Large, complex projects are likely to have more requirements. Smaller, quicker projects will have fewer requirements and will need shorter requirements documents, which can double as project proposals.

Why Write Requirements Documents?

Although writing a complete requirements document is time-consuming, there are many advantages to having one—Involving major stakeholders in writing requirements helps to ensure that everyone agrees on what is to be done. This can avert misunderstandings down the road and save time that later might be wasted in rework. You can use a requirements document to help set expectations about what will and will not be accomplished in the current development cycle.

A solid requirements document also serves as a guide to development and testing throughout the project. If you have a list of everything your product should do, it's easier to create a product that does everything you wanted (and to test it to see if it really does it all). Your requirements document can also help you schedule time and resources more efficiently and to plan your project more accurately. You'll also have the benefit of knowing when you're done with development.

Here are a few of the problems which can be avoided (or at least lessened) by having a good requirements document:

building to requirements which do not really reflect the needs of stakeholders building a project from inconsistent or incomplete requirements making changes to requirements during development, which is costly misunderstandings between customers or end users and developers which result in a product

that is not what was wanted forgetting plans for the project feature creep

How Are Requirements Documents Prepared?

A lot of work happens before a requirements document is written. Your project will benefit from the time you spend finding out what the requirements are before writing them down. Once you have gathered and recorded requirements, they should be classified and prioritized. With the list of prioritized requirements and any other project documents you already have, you will be able to compile the requirements document.

Gathering Requirements

Requirements, as stated earlier, should come from end users, customers, and sometimes developers. Most requirements should come from end users. Identify your target user population, and then interview some people from that population. It really is important to talk to people from the target population who are not in any way associated with the project; they will think of things you cannot even

imagine. If possible, observe people using something similar to your intended product, and take notes on what works and what doesn't. Make your own observations as well as asking their opinions. Describe your intended product and ask people how they would use it, what features they would need, and what features they would find less useful. You will get some good data this way; just keep in mind that users are generally not developers, and they are not good at describing features they would want before they actually sit down to use a product. However, you can correct for this later during requirements validation.

Interview your customers (the people who are paying for the project) to find out what their priorities are. These will sometimes conflict with user and developer priorities. Record them all and negotiate during the prioritizing phase of requirements gathering.

Other methods for gathering requirements include:

writing personas and scenarios Examining project-related documents like proposals, communications, development

agreements, etc. interviewing developers and designers researching the market to find similar products and noting what works/doesn't work about

these doing task analysis, if the product you are developing facilitates a current tasks users are doing

Recording Requirements

As you gather requirements, use a form to capture them. This will help your requirements remain consistent in form and in terms of the information you capture for each. Basic information to collect:

A short title A detailed description of the requirement The person recording the requirement The source of the requirement (a scenario, a user interview, the project proposal, etc.) A rationale: why is this requirement necessary? A list of the stakeholders most affected by this requirement Initial desired priority, according to the source of the requirement

You may wish to record additional information depending on your project's needs. If your development team is distributed, consider using a web-based requirements tracking system which allows you to record, classify, and prioritize requirements, which are stored in a database.

Classifying Requirements

Once requirements are recorded, the development team should spend some time classifying them. For each requirement, identify the following attributes:

Project component (system, user interface, database, security…) type of requirement (system, process, out of scope) system components, if applicable (for more complex projects) volatility and clarity of requirement

requirement, assumption, or issue

Project component refers to the part of the product that the requirement affects. For example, a website that dynamically draws information from a database to populate web page templates would have components such as templates, database, user interface, and business logic (or code). Some requirements are applicable to more than one project component. Type of requirement identifies whether the recorded requirement is something that you need to deal with or not: system requirements have to do with how the system is built or functions, so you'll want to deal with those. Process requirements have to do with how users use the system; you can't control these. For example, the user must connect headphones to his computer is a process requirement for most multimedia projects. You can't ensure that users always do this, so don't include it in your list of system requirements. Out-of-scope requirements are those that are impossible or unfeasible to implement during the current development cycle.

Don't throw away process or out-of-scope requirements. Process requirements can inform your development; can you build your product so that it doesn't matter whether the user has headphones or not? Out-of-scope requirements are useful for future project proposals for later versions. Keep both in a separate list so you know where you've been.

System components are applicable for projects that have several different parts. For example, if your project includes an authoring tool that allows nontechnical users to enter information which is then bundled and published as, say, a web page, your system components would include at least authoring tool, templates, and publishing logic. These can be combined with project components for some products.

Volatility refers to how likely a requirement is to change; knowing which are stable and which are less so is very useful. Clarity refers to how clearly the requirement is written. Those that are unclear should be refined before the document is completed.

Finally, identify whether what you have recorded is really a requirement, or if it is an assumption or an issue. Assumptions are facts which are simply true; if your project is web-based, you can assume that your product will be housed on a web server of some kind. You should still state this in your document. Sometimes the line between assumptions and requirements is hazy. Issues are open-ended questions that should be resolved at some point. They may result in requirements or assumptions, or they may be resolved some other way. Flag them as issues to avoid overlooking them.

Once your requirements have been classified, you can arrange them in groups of related requirements. This makes it easier to prioritize them, and gives you a better understanding of which parts of your project will be the most complicated. Typical groupings are by system or project component, with process requirements, out-of-scope requirements, assumptions, and issues pulled out into separate lists.

Prioritizing Requirements

Although your requirements were prioritized when they were recorded, you will likely find that you have more high-priority requirements than development resources. There are probably also requirements in your list which simply should not be implemented, but which seemed desirable when they were proposed.

A second pass at prioritizing will help you select which requirements will be implemented now and which need to be removed to the next version of your project. Prioritization helps to ensure the finished product is as close as possible to what everyone wanted in the first place.

Ideally, prioritization is conducted with representatives of all major stakeholders present, including developers and project managers, end users, and customers. The process of prioritization is a negotiation where all parties discuss their needs and make compromises so that a robust, useful product can be developed on time and on budget. Each requirement is considered and several factors are balanced to assign a final priority:

desired priority, according to source which stakeholders are affected how costly the requirement is to implement whether the requirement is technologically feasible whether the requirement is clearly understood

In reality, the second round of prioritization often happens when developers look at a list of requirements and start identifying those which are especially high-cost or high-risk. It is wise to involve stakeholders other than developers at this stage, though. You may find that a certain high-cost requirement is so important that stakeholders are willing to give up several smaller requirements in order to have it implemented. As you can see, prioritization has an impact on project scheduling.

Once the requirements have been prioritized, you are ready to assemble the requirements document. Another round of prioritization may occur once the first draft is written, so be prepared to go through this process again.

Writing the Document

Armed with your list of prioritized, classified requirements, along with any other pertinent project related documents; you are ready to write the requirements document. The good news is that at this point, most of it is written already.The Introduction

The document should start with an introduction which explains the purpose of the requirements document and how to use it. State any technical background that is needed to understand the requirements and point less-technical readers to overview sections which will be easier to understand. Explain the scope of the product to define boundaries and set expectations.

Include the business case for the product (why is it a good idea?). Finally, include a summary or a list of the most important requirements, to give a general understanding of the requirements and focus attention on the most critical ones.

General Description

Devote the next section of the document to a description of the product in nontechnical terms. This is valuable for readers who want to know what the project is about without wading through the technical requirements. Include a narrative which gives readers an idea of what the product will do. Identify the product perspective: the need(s) the product will serve, the primary stakeholders, and the developers.

Describe the main functions of the product (what activities will users be able to do with it?) and describe typical user characteristics. Explain any constraints you will face during development, and list your assumptions and any technical dependencies.

Specific Requirementso Include your requirements list here, divided into whatever groupings are convenient for

you.

Appendiceso Include, if you wish, any source documents such as personas and scenarios, transcripts

of user interviews, and so on.

Glossaryo Explain any unusual terms, acronyms, or abbreviations (either technical or domain

specific).

Referenceso List references and source documents

Indexo If your document is long, consider including an index.

How Is a Requirements Document Used?

By this point in the process, you should have clarified what your product will actually be like. If you were careful to include them in the process of gathering requirements, you should have buy in from all your major stakeholder groups. Perhaps some of your findings were surprising; hopefully you were able to incorporate the surprises into your project plan.

Once the first draft of your requirements document is complete, don't file it and forget it. Part of the point of writing one is the process of unearthing, recording, organizing, and prioritizing requirements, but the document is useful during development as well. First, however, it should be validated, and probably revised.

Validating Requirements

Go back to end-users and customers with the draft requirements document and have them read it (mostly the narrative sections). Ask if this is what they had in mind, or if they think any changes are in order. It's much easier to make changes now, while your project is still words on paper, than after development has begun. Take their comments and make changes; you may need to revisit the previous stages (recording, organizing, and prioritizing), but it will be easier this time.

The vital thing is to come up with a requirements specification that meets the needs of the major stakeholders. Repeat the process until there is agreement, and get representatives of each stakeholder group to actually sign off on the document.

During Product Development

During development, refer to the document for guidance. Interface designers, instructional designers, and developers should use it as a map to direct their efforts. If someone's work is called into question,

check with the requirements document to decide which direction to proceed. This isn't to say that you have to treat it as inviolate. It's possible that small requirements changes will continue to occur. Just be sure to get stakeholder agreement before altering the specification.

During Product Testing

Continue to check the ongoing development process against the requirements; does the emerging product do what it ought? When the product is in testing, check it against the requirements again. Was anything omitted? If you're having problems with a feature, check with the requirements document to see if there is a way around the problem. Sometimes developers add features which don't need to be there, and in a time or budget crunch, you can use the requirements specification as a reason to put off an extra feature.

After Release

Keep track of the "wish list" and "version two" requirements. You may have the makings of a grant proposal there!

SPECIFICATION DOCUMENTA specification document in systems engineering and software development is the documentation that describes the requested behavior of an engineering system. The documentation typically describes what is needed by the system user as well as requested properties of inputs and outputs (e.g. of the software system). A functional specification is the more technical response to a matching requirements document, e.g. the Product Requirement Document. Thus it picks up the results of the requirements analysis stage. On more complex systems multiple levels of functional specifications will typically nest to each other, e.g. on the system level, on the module level and on the level of technical details.

In systems engineering a functional specification is a document that clearly and accurately describes the essential technical requirements for items, materials, or services including the procedures by which it can be determined that the requirements have been met. Specifications help avoid duplication and inconsistencies, allow for accurate estimates of necessary work and resources, act as a negotiation and reference document for engineering changes, provide documentation of configuration, and allow for consistent communication among those responsible for the eight primary functions of Systems Engineering. They provide a precise idea of the problem to be solved so that they can efficiently design the system and estimate the cost of design alternatives. They provide guidance to testers for verification (qualification) of each technical requirement.

A functional specification does not define the inner workings of the proposed system; it does not include the specification of how the system function will be implemented. Instead, it focuses on what various outside agents (people using the program, computer peripherals, or other computers, for example) might "observe" when interacting with the system. A typical functional specification might state as follows:

When the user clicks the OK button, the dialog is closed and the focus is returned to the main window in the state it was in before this dialog was displayed.

Such a requirement describes an interaction between an external agent (the user) and the software system. When the user provides input to the system by clicking the OK button, the program responds (or should respond) by closing the dialog window containing the OK button.

It can be informal, in which case it can be considered as a blueprint or user manual from a developer point of view, or formal, in which case it has a definite meaning defined in mathematical or programmatic terms. In practice, most successful specifications are written to understand and fine-tune applications that were already well-developed, although safety-critical software systems are often carefully specified prior to application development. Specifications are most important for external interfaces that must remain stable.

Purpose

There are many purposes for functional specifications. One of the primary purposes on team projects is to achieve some form of team consensus on what the program is to achieve before making the more time-consuming effort of writing source code and test cases, followed by a period of debugging. Typically, such consensus is reached after one or more reviews by the stakeholders on the project at hand after having negotiated a cost-effective way to achieve the requirements the software needs to fulfill.

To let the developers know what to build

To let the testers know what tests to run

To let stakeholders know what they are getting

Process

In the ordered industrial software engineering life-cycle (waterfall model), functional specification describes what has to be implemented. The next, Systems architecture document describes how the functions will be realized using a chosen software environment. In non-industrial, prototypical systems development, functional specifications are typically written after or as part of requirements analysis.

When the team agrees that functional specification consensus is reached, the functional spec is typically declared "complete" or "signed off". After this, typically the software development and testing team write source code and test cases using the functional specification as the reference. While testing is performed, the behavior of the program is compared against the expected behavior as defined in the functional specification.

Methods

One popular method of writing a functional specification document involves drawing or rendering either simple wireframes or accurate, graphically designed UI screenshots. After this has been completed, and the screen examples are approved by all stakeholders, graphical elements can be numbered and written instructions can be added for each number on the screen example. For example, a login screen can have the username field labeled '1' and password field labeled '2,' and then each number can be declared in writing, for use by software engineers and later for beta testing purposes to ensure that functionality is as intended. The benefit of this method is that countless additional details can be attached to the screen examples.

TEST PLANA test plan is a document detailing the objectives, target market, internal beta team, and processes for a specific beta test for a software or hardware product. The plan typically contains a detailed understanding of the eventual workflow.

A test plan documents the strategy that will be used to verify and ensure that a product or system meets its design specifications and other requirements. A test plan is usually prepared by or with significant input from test engineers.

Depending on the product and the responsibility of the organization to which the test plan applies, a test plan may include a strategy for one or more of the following:

Design Verification or Compliance test - to be performed during the development or approval stages of the product, typically on a small sample of units.

Manufacturing or Production test - to be performed during preparation or assembly of the product in an ongoing manner for purposes of performance verification and quality control.

Acceptance or Commissioning test - to be performed at the time of delivery or installation of the product.

Service and Repair test - to be performed as required over the service life of the product.

Regression test - to be performed on an existing operational product, to verify that existing functionality didn't get broken when other aspects of the environment are changed (e.g., upgrading the platform on which an existing application runs).

A complex system may have a high level test plan to address the overall requirements and supporting test plans to address the design details of subsystems and components.

Test plan document formats can be as varied as the products and organizations to which they apply. There are three major elements that should be described in the test plan: Test Coverage, Test Methods, and Test Responsibilities. These are also used in a formal test strategy.

Test coverage

Test coverage in the test plan states what requirements will be verified during what stages of the product life. Test Coverage is derived from design specifications and other requirements, such as safety standards or regulatory codes, where each requirement or specification of the design ideally will have one or more corresponding means of verification. Test coverage for different product life stages may overlap, but will not necessarily be exactly the same for all stages. For example, some requirements may be verified during Design Verification test, but not repeated during Acceptance test. Test coverage also feeds back into the design process, since the product may have to be designed to allow test access (see Design for Test).

Test methods

Test methods in the test plan state how test coverage will be implemented. Test methods may be determined by standards, regulatory agencies, or contractual agreement, or may have to be created new. Test methods also specify test equipment to be used in the performance of the tests and establish pass/fail criteria. Test methods used to verify hardware design requirements can range from very simple steps, such as visual inspection, to elaborate test procedures that are documented separately.

Test responsibilities

Test responsibilities include what organizations will perform the test methods and at each stage of the product life. This allows test organizations to plan, acquire or develop test equipment and other resources necessary to implement the test methods for which they are responsible. Test responsibilities also includes, what data will be collected, and how that data will be stored and reported (often referred to as "deliverables"). One outcome of a successful test plan should be a record or report of the verification of all design specifications and requirements as agreed upon by all parties.

IEEE 829 test plan structure

IEEE 829-2008, also known as the 829 Standard for Software Test Documentation, is an IEEE standard that specifies the form of a set of documents for use in defined stages of software testing, each stage potentially producing its own separate type of document.[1] These stages are:

Test plan identifier

Introduction

Test items

Features to be tested

Features not to be tested

Approach

Item pass/fail criteria

Suspension criteria and resumption requirements

Test deliverables

Testing tasks

Environmental needs

Responsibilities

Staffing and training needs

Schedule

Risks and contingencies

Approvals

The IEEE documents that suggest what should be contained in a test plan are:

829-2008 IEEE Standard for Software and System Test Documentation

829-1998 IEEE Standard for Software Test Documentation (superseded by 829-2008)

829-1983 IEEE Standard for Software Test Documentation (superseded by 829-1998)

1008-1987 IEEE Standard for Software Unit Testing

1012-2004 IEEE Standard for Software Verification and Validation

1012-1998 IEEE Standard for Software Verification and Validation (superseded by 1012-2004)

1012-1986 IEEE Standard for Software Verification and Validation Plans (superseded by 1012-1998)

1059-1993 IEEE Guide for Software Verification & Validation Plans (withdrawn)

Write a sample Test Plan

TEST CASEA test case, in software engineering, is a set of conditions under which a tester will determine whether an application, software system or one of its features is working as it was originally established for it to do. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test oracle. In some settings, an oracle could be a requirement or use case, while in others it could be a heuristic. It may take many test cases to determine that a software program or system is considered sufficiently scrutinized to be released. Test cases are often referred to as test scripts, particularly when written - when they are usually collected into test suites.

Formal test cases

In order to fully test that all the requirements of an application are met, there must be at least two test cases for each requirement: one positive test and one negative test. If a requirement has sub-requirements, each sub-requirement must have at least two test cases. Keeping track of the link between the requirement and the test is frequently done using a traceability matrix. Written test cases should include a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted.

A formal written test-case is characterized by a known input and by an expected output, which is worked out before the test is executed. The known input should test a precondition and the expected output should test a post condition.

Informal test cases

For applications or systems without formal requirements, test cases can be written based on the accepted normal operation of programs of a similar class. In some schools of testing, test cases are not written at all but the activities and results are reported after the tests have been run.

In scenario testing, hypothetical stories are used to help the tester think through a complex problem or system. These scenarios are usually not written down in any detail. They can be as simple as a diagram for a testing environment or they could be a description written in prose. The ideal scenario test is a story that is motivating, credible, complex, and easy to evaluate. They are usually different from test cases in that test cases are single steps while scenarios cover a number of steps of the key.

Typical written test case format

A test case is usually a single step, or occasionally a sequence of steps, to test the correct behavior/functionality, features of an application. An expected result or expected outcome is usually given.

Additional information that may be included:

test case ID

test case description

test step or order of execution number

related requirement(s)

depth

test category

author

check boxes for whether the test can be or has been automated

pass/fail

remarks

Larger test cases may also contain prerequisite states or steps, and descriptions.

A written test case should also contain a place for the actual result.

These steps can be stored in a word processor document, spreadsheet, database or other common repository.

In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table.

Test suites often also contain:

Test summary

Configuration

Besides a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted, the most time consuming part in the test case is creating the tests and modifying them when the system changes.

Under special circumstances, there could be a need to run the test, produce results, and then a team of experts would evaluate if the results can be considered as a pass. This happens often on new products' performance number determination. The first test is taken as the base line for subsequent test / product release cycles.

Acceptance tests, which use a variation of a written test case, are commonly performed by a group of end-users or clients of the system to ensure the developed system meets the requirements specified or the contract. User acceptance tests are differentiated by the inclusion of happy path or positive test cases to the almost complete exclusion of negative test cases.

What’s a Bug?A software bug is an error, flaw, failure, or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program's source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code. A program that contains a large number of bugs, and/or bugs that seriously interfere with its functionality, is said to be buggy. Reports detailing bugs in a program are commonly known as bug reports, defect reports, fault reports, problem reports, trouble reports, change requests, and so forth.

Bugs trigger errors that can in turn have a wide variety of ripple effects, with varying levels of inconvenience to the user of the program. Some bugs have only a subtle effect on the program's functionality, and may thus lie undetected for a long time. More serious bugs may cause the program to crash or freeze. Others qualify as security bugs and might for example enable a malicious user to bypass access controls in order to obtain unauthorized privileges.

The results of bugs may be extremely serious. Bugs in the code controlling the Therac-25 radiation therapy machine were directly responsible for some patient deaths in the 1980s. In 1996, the European Space Agency's US$1 billion prototype Ariane 5 rocket had to be destroyed less than a minute after launch, due to a bug in the on-board guidance computer program. In June 1994, a Royal Air Force Chinook helicopter crashed into the Mull of Kintyre, killing 29. This was initially dismissed as pilot error, but an investigation by Computer Weekly uncovered sufficient evidence to convince a House of Lords inquiry that it may have been caused by a software bug in the aircraft's engine control computer.

In 2002, a study commissioned by the US Department of Commerce' National Institute of Standards and Technology concluded that "software bugs, or errors, are so prevalent and so detrimental that they cost the US economy an estimated $59 billion annually, or about 0.6 percent of the gross domestic product".

Etymology

Use of the term "bug" to describe inexplicable defects has been a part of engineering jargon for many decades and predates computers and computer software; it may have originally been used in hardware engineering to describe mechanical malfunctions. For instance, Thomas Edison wrote the following words in a letter to an associate in 1878:

It has been just so in all of my inventions. The first step is an intuition, and comes with a burst, then difficulties arise — this thing gives out and [it is] then that "Bugs" — as such little faults and difficulties are called — show themselves and months of intense watching, study and labor are requisite before commercial success or failure is certainly reached.

Baffle Ball, the first mechanical pinball game, was advertised as being "free of bugs" in 1931. Problems with military gear during World War II were referred to as bugs (or glitches).

A page from the Harvard Mark II electromechanical computer's log, featuring a dead moth that was removed from the device

The term "bug" was used in an account by computer pioneer Grace Hopper, who publicized the cause of a malfunction in an early electromechanical computer. A typical version of the story is given by this quote:

In 1946, when Hopper was released from active duty, she joined the Harvard Faculty at the Computation Laboratory where she continued her work on the Mark II and Mark III. Operators traced an error in the Mark II to a moth trapped in a relay, coining the term bug. This bug was carefully removed and taped to the log book. Stemming from the first bug, today we call errors or glitches in a program a bug.

Hopper was not actually the one who found the insect, as she readily acknowledged. The date in the log book was September 9, 1947, although sometimes erroneously reported as 1945. The operators who did find it, including William "Bill" Burke, later of the Naval Weapons Laboratory, Dahlgren, Virginia, were familiar with the engineering term and, amused, kept the insect with the notation "First actual case of bug being found." Hopper loved to recount the story. This log book, complete with attached moth, is part of the collection of the Smithsonian National Museum of American History, though it is not currently on display.

The related term "debug" also appears to predate its usage in computing: the Oxford English Dictionary's etymology of the word contains an attestation from 1945, in the context of aircraft engines.

Prevalence

In software development projects, a "mistake" or "fault" can be introduced at any stage during development. Bugs are a consequence of the nature of human factors in the programming task. They arise from oversights or mutual misunderstandings made by a software team during specification, design, coding, data entry and documentation. For example, in creating a relatively simple program to sort a list of words into alphabetical order, one's design might fail to consider what should happen when a word contains a hyphen. Perhaps, when converting the abstract design into the chosen programming language, one might inadvertently create an off-by-one error and fail to sort the last word in the list. Finally, when typing the resulting program into the computer, one might accidentally type a "<" where a ">" was intended, perhaps resulting in the words being sorted into reverse alphabetical order.

Another category of bug is called a race condition that can occur when programs have multiple components executing at the same time, either on the same system or across multiple systems interacting across a network. If the components interact in a different order than the developers intended, it may break the logical flow of the program. These bugs can be difficult to detect or anticipate, since they may not occur during every execution of a program.

More complex bugs can arise from unintended interactions between different parts of a computer program. This frequently occurs because computer programs can be complex — millions of lines long in some cases — often having been programmed by many people over a great length of time, so that programmers are unable to mentally track every possible way in which parts can interact.

Mistake metamorphism

There is ongoing debate over the use of the term "bug" to describe software errors. One argument is that the word "bug" is divorced from a sense that a human being caused the problem, and instead implies that the defect arose on its own, leading to a push to abandon the term "bug" in favor of terms such as "defect", with limited success.

In software engineering, mistake metamorphism (from Greek Meta = "change", morph = "form") refers to the evolution of a defect in the final stage of software deployment. Transformation of a "mistake" committed by an analyst in the early stages of the software development lifecycle, which leads to a "defect" in the final stage of the cycle, has been called 'mistake metamorphism'.

Different stages of a "mistake" in the entire cycle may be described by using the following terms:

Mistake

Anomaly

Fault

Failure

Error

Exception

Crash

Bug

Defect

Incident

Side Effect

Prevention

The software industry has put much effort into finding methods for preventing programmers from inadvertently introducing bugs while writing software. These include:

Programming style

While typos in the program code are often caught by the compiler, a bug usually appears when the programmer makes a logic error. Various innovations in programming and defensive programming are designed to make these bugs less likely or easier to spot. In some programming languages, so-called typos, especially of symbols or logical/mathematical operators, actually represent logic errors, since the mistyped constructs are accepted by the compiler with a meaning other than that which the programmer intended.

Programming techniques

Bugs often create inconsistencies in the internal data of a running program. Programs can be written to check the consistency of their own internal data while running. If an inconsistency is encountered, the program can immediately halt, so that the bug can be located and fixed. Alternatively, the program can simply inform the user, attempt to correct the inconsistency, and continue running.

Development methodologies

There are several schemes for managing programmer activity, so that fewer bugs are produced. Many of these fall under the discipline of software engineering (which addresses software design issues as well). For example, formal program specifications are used to state the exact behavior of programs, so that design bugs can be eliminated. Unfortunately, formal specifications are impractical or impossible for anything but the shortest programs, because of problems of combinatorial explosion and indeterminacy.

In modern times, popular approaches include automated unit testing and automated acceptance testing (sometimes going to the extreme of test-driven development), and agile software development (which is often combined with, or even in some cases mandates, automated testing). All of these approaches are supposed to catch bugs and poorly-specified requirements soon after they are introduced, which should make them easier and cheaper to fix, and to catch at least some of them before they enter into production use.

Programming language support

Programming languages often include features which help programmers prevent bugs, such as static type systems, restricted namespaces and modular programming, among others. For example, when a programmer writes (pseudocode) LET REAL_VALUE PI = "THREE AND A BIT", although this may be syntactically correct, the code fails a type check. Depending on the language and implementation, this may be caught by the compiler or at run-time. In addition, many recently invented languages have deliberately excluded features which can easily lead to bugs, at the expense of making code slower than it need be: the general principle being that, because of Moore's law, computers get faster and software engineers get slower; it is almost always better to write simpler, slower code than "clever", inscrutable code, especially considering that maintenance cost is substantial. For example, the Java programming language does not support pointer arithmetic; implementations of some languages such as Pascal and scripting languages often have runtime bounds checking of arrays, at least in a debugging build.

Code analysis

Tools for code analysis help developers by inspecting the program text beyond the compiler's capabilities to spot potential problems. Although in general the problem of finding all programming errors given a specification is not solvable (see halting problem), these tools exploit the fact that human programmers tend to make the same kinds of mistakes when writing software.

Instrumentation

Tools to monitor the performance of the software as it is running, either specifically to find problems such as bottlenecks or to give assurance as to correct working, may be embedded in the code explicitly (perhaps as simple as a statement saying PRINT "I AM HERE"), or provided as tools. It is often a surprise to find where most of the time is taken by a piece of code, and this removal of assumptions might cause the code to be rewritten.

Debugging

A new bug submitted by the user is unconfirmed. Once it has been reproduced by a developer, it is a confirmed bug. The confirmed bugs are later fixed. Bugs belonging to other categories (unreproducible, will not be fixed, etc.) are usually in the minority

Finding and fixing bugs, or "debugging", has always been a major part of computer programming. Maurice Wilkes, an early computing pioneer, described his realization in the late 1940s that much of the rest of his life would be spent finding mistakes in his own programs. As computer programs grow more complex, bugs become more common and difficult to fix. Often programmers spend more time and effort finding and fixing bugs than writing new code. Software testers are professionals whose primary task is to find bugs, or write code to support testing. On some projects, more resources can be spent on testing than in developing the program.

Usually, the most difficult part of debugging is finding the bug in the source code. Once it is found, correcting it is usually relatively easy. Programs known as debuggers exist to help programmers locate bugs by executing code line by line, watching variable values, and other features to observe program behavior. Without a debugger, code can be added so that messages or values can be written to a console (for example with printf in the C programming language) or to a window or log file to trace program execution or show values.

However, even with the aid of a debugger, locating bugs is something of an art. It is not uncommon for a bug in one section of a program to cause failures in a completely different section, thus making it especially difficult to track (for example, an error in a graphics rendering routine causing a fileI/O routine to fail), in an apparently unrelated part of the system.

Sometimes, a bug is not an isolated flaw, but represents an error of thinking or planning on the part of the programmer. Such logic errors require a section of the program to be overhauled or rewritten. As a part of Code review, stepping through the code modelling the execution process in one's head or on paper can often find these errors without ever needing to reproduce the bug as such, if it can be shown there is some faulty logic in its implementation.

But more typically, the first step in locating a bug is to reproduce it reliably. Once the bug is reproduced, the programmer can use a debugger or some other tool to monitor the execution of the program in the faulty region, and find the point at which the program went astray.

It is not always easy to reproduce bugs. Some are triggered by inputs to the program which may be difficult for the programmer to re-create. One cause of theTherac-25 radiation machine deaths was a bug (specifically, a race condition) that occurred only when the machine operator very rapidly entered a

treatment plan; it took days of practice to become able to do this, so the bug did not manifest in testing or when the manufacturer attempted to duplicate it. Other bugs may disappear when the program is run with a debugger; these are heisenbugs (humorously named after the Heisenberg uncertainty principle).

Debugging is still a tedious task requiring considerable effort. Since the 1990s, particularly following the Ariane 5 Flight 501 disaster, there has been a renewed interest in the development of effective automated aids to debugging. For instance, methods of static code analysis by abstract interpretation have already made significant achievements, while still remaining much of a work in progress.

As with any creative act, sometimes a flash of inspiration will show a solution, but this is rare and, by definition, cannot be relied on.

There are also classes of bugs that have nothing to do with the code itself. If, for example, one relies on faulty documentation or hardware, the code may be written perfectly properly to what the documentation says, but the bug truly lies in the documentation or hardware, not the code. However, it is common to change the code instead of the other parts of the system, as the cost and time to change it is generally less. Embedded systems frequently have workarounds for hardware bugs, since to make a new version of a ROM is much cheaper than remanufacturing the hardware, especially if they are commodity items.

Bug management

Bug management encompasses more than bug tracking, and there exists no industry-wide standard. Proposed changes to software – bugs as well as enhancement requests and even entire releases – are commonly tracked and managed using bug tracking systems. The items added may be called defects, tickets, issues, or, following the agile development paradigm, stories and epics. The systems allow or even require some type of categorization of each issue. Categories may be objective, subjective or a combination, such as version number, area of the software, severity and priority, as well as what type of issue it is, such as a feature request or a bug. Issue tracking systems may also be used, but are more commonly used for tracking external issues such as customer trouble reports.

It is common practice for software to be released with known bugs that are considered "non-critical" as defined by the software producer(s). While software products may, by definition, contain any number of unknown bugs, measurements during testing can provide an estimate of the number of likely bugs remaining; this becomes more reliable the longer a product is tested and developed. Most big software projects maintain two lists of "known bugs"— those known to the software team, and those to be told to users. The second list informs users about bugs that are not fixed in the current release, or not fixed at all, and a workaround may be offered.

In the case of software bugs, "severity" can be an attempt to categorize the symptom or actual behavior of the bug. Severity levels are not standardized and are decided by each software producer, if they are even used. For example, bug severity levels might be "crash or hang", "no workaround" (meaning there

is no way the customer can accomplish a given task), "has workaround" (meaning there is a way for the user to recover and accomplish the task), "UI" or "visual defect" (for example, a missing image or displaced button or form element), or "documentation error". Some software publishers use more qualified severities such as "critical", "high", "low," "blocker," or "trivial".The severity of a bug may be a separate category to its priority for fixing, and the two may be quantified and managed separately. For example, if a flaw is found in an application which causes it to crash, yet the crash is so rare and takes, say, ten extremely unusual or unlikely steps to produce it, management may set its priority as "low" or even "will not fix." How the priority for fixing is used is decided internally by each software producer. Priorities are sometimes numerical and sometimes words, such "critical," "high," "low" or "deferred"; note that these can be similar or even identical to severity ratings when looking at different software producers. For example, a software company may decide that priority 1 bugs are always to be fixed for the next release, whereas "5" could mean its fix is put off – sometimes indefinitely.

A software publisher may opt not to fix a particular bug for a number of reasons, including:

A deadline must be met and priorities are such that all but those above a certain severity are fixed for the current software release.

The bug is already fixed in an upcoming release, and it is not serious enough to warrant an immediate update or patch

The changes to the code required to fix the bug are too costly, will take too long for the current release, or affect too many other areas of the software.

Users may be relying on the undocumented, buggy behavior; it may introduce a breaking change.

The problem is in an area which will be obsolete with an upcoming release; fixing it is unnecessary.

It's "not a bug". A misunderstanding has arisen between expected and perceived behavior, when such misunderstanding is not due to confusion arising from design flaws, or faulty documentation.

The amount and type of damage a software bug can cause naturally affects decision-making, processes and policy regarding software quality. In applications such as manned space travel or automotive safety, since software flaws have the potential to cause human injury or even death, such software will have far more scrutiny and quality control than, for example, an online shopping website. In applications such as banking, where software flaws have the potential to cause serious financial damage to a bank or its customers, quality control is also more important than, say, a photo editing application. NASA's SATC managed to reduce the number of errors to less than 0.1 per 1000 lines of code (SLOC) but this was not felt to be feasible for projects in the business world.

A school of thought popularized by Eric S. Raymond as Linus's Law says that popular open-source software has more chance of having few or no bugs than other software, because "given enough

eyeballs, all bugs are shallow". This assertion has been disputed, however: computer security specialist Elias Levy wrote that "it is easy to hide vulnerabilities in complex, little understood and undocumented source code," because, "even if people are reviewing the code, that doesn't mean they're qualified to do so."

Security vulnerabilities

Malicious software may attempt to exploit known vulnerabilities in a system–which may or may not be bugs. Viruses are not bugs in themselves–they are typically programs that are doing precisely what they were designed to do. However, viruses are occasionally referred to as such in the popular press. In addition, it is often a security bug in a computer program that allows viruses to work in the first place.

Common types of computer bugs

Conceptual error

Code is syntactically correct, but the programmer or designer intended it to do something else).

Arithmetic bugs

Division by zero.

Arithmetic overflow or underflow.

Loss of arithmetic precision due to rounding or numerically unstable algorithms.

Logic bugs

Infinite loops and infinite recursion.

Off-by-one error, counting one too many or too few when looping.

Syntax bugs

Use of the wrong operator, such as performing assignment instead of equality test. For example, in some languages x=5 will set the value of x to 5 while x==5 will check whether x is currently 5 or some other number. In simple cases often the compiler can generate a warning. In many languages, the language syntax is deliberately designed to guard against this error.

Resource bugs

Null pointer dereference.

Using an uninitialized variable.

Using an otherwise valid instruction on the wrong data type (see packed decimal/binary coded decimal).

Access violations.

Resource leaks, where a finite system resource (such as memory or file handles) becomes exhausted by repeated allocation without release.

Buffer overflow, in which a program tries to store data past the end of allocated storage. This may or may not lead to an access violation or storage violation. These bugs can form security vulnerability.

Excessive recursion which — though logically valid — causes stack overflow.

Use-after-free error, where a pointer is used after the system has freed the memory it references.

Double free error.

Multi-threading programming bugs

Deadlock, where task A can't continue until task B finishes, but at the same time, task B can't continue until task a finishes.

Race condition, where the computer does not perform tasks in the order the programmer intended.

Concurrency errors in critical sections, mutual exclusions and other features of concurrent processing. Time-of-check-to-time-of-use (TOCTOU) is a form of unprotected critical section.

Interfacing bugs

Incorrect API usage.

Incorrect protocol implementation.

Incorrect hardware handling.

Incorrect assumptions of a particular platform.

Incompatible systems. Often a proposed "new API" or new communications protocol may seem to work when both computers use the old version or both computers use the new version, but upgrading only the receiver exposes backward compatibility problems; in other cases upgrading only the transmitter exposes forward compatibility problems. Often it is not feasible to upgrade every computer simultaneously—in particular, in the telecommunication industry or the internet. Even when it is feasible to update every computer simultaneously, sometimes people accidentally forget to update every computer—the Knight involved one such incompatibility between the old API and a new API.

Performance bugs

Too high computational complexity of algorithm.

Random disk or memory access.

Team working bugs

Unpropagated updates; e.g. programmer changes "myAdd" but forgets to change "mySubtract", which uses the same algorithm. These errors are mitigated by the Don't Repeat Yourself philosophy.

Comments out of date or incorrect: many programmers assume the comments accurately describe the code.

Differences between documentation and the actual product.

TYPES OF SOFTWARE1. Web Sites

2. Web Applications

3. Mobile Applications

4. Mobile Platforms - Android, iOS, Windows, Kindles, Firefox OS

TESTING TOOLS1. Web Developer tools

2. Bug tracking tools

3. Fiddler

4. Visual Studio Team Foundation

5. Jira

6. Asana

7. Selenium