Do not automate GUI

Reinvention of wheel (HTTP performance testing) like an effective tool for regression testing

Atila Inovecký

Something about pyramidsWHY to do API Testing

Don't reinvent a wheel

Abraham Maslow – Human Needs

Don't reinvent a wheel

Gojko Adzic - Quality Criteria

Don't reinvent a wheel

Mike Cohn (2009)– Automated regression testing

Don't reinvent a wheel

API testing in 2001• Kaner, Bach, Pettichord: Lessons Learned in Software Testing – Lesson 132:

Automate tests using programming interfaces“…When we look at many product testing effort we’ve observed, we conclude that there is a strong correlation between the availability of programming interfaces for testing and the development of powerful automated test suites. Programming interfaces include APIs, command-line interfaces, COM interfaces, HTTP and more. …If you’re serious about automation, you’ll learn or get someone who knows this to help..…Focus your attention on where automation can help the most. Sometimes, you can effectively automate the GUI; other times you can’t. Don’t let people’s preconceptions of what automation has to look like limit you.”

Don't reinvent a wheel

Ice Cream Cone Antipattern

Don't reinvent a wheel

Reasons for antipattern(traditional approach)• Automate everything!• Just QA is responsible for quality• Few testing rounds – no Continuous Integration• WHAT TO TEST(Business Functionality)=HOW TO TEST (GUI)?• “This small hotfix won’t break anything”• Production delivery 1 per year• DIDN’T FAIL YET• Trying to find a lot bugs

Don't reinvent a wheel

Test find bugs, not automation• Value of the regression test depend on value of the covered

functionality• Count of found bugs say nothig about value\effectivity of automated

test• Most of regression bugs is caused by unexpected impact of backend

integration not GUI• Automation is not medicine for wrong communication

Don't reinvent a wheel

Conclusions

• Unit testing is the core of regression testing• API testing is not the FUTURE but it is really just a reinvented wheel• Future of automated GUI testing is (Exploratory tests, …)???• GUI testing vs API testing

Don't reinvent a wheel

Something about REST API

How to test API through HTTP (REST)

Don't reinvent a wheel

REST – Representational State Transfer• REST or RESTful web services are one way of providing

interoperability between computer systems on the internet. REST-compliant web services allow requesting systems to access and manipulate textual representations of web resources using a uniform and predefined set of stateless operations.

Don't reinvent a wheel

Layered architecture of RESTful interactions

Don't reinvent a wheel

RESTful URLs• URLs intended to improve the usability and accessibility of a website

or web service by being immediately and intuitively meaningful to non-expert users.

Non-semantic URL Semantic URL

http://example.com/index.php?page=name http://example.com/name

http://example.com/index.php?page=consulting/marketing http://example.com/consulting/marketing

http://example.com/products?category=2&pid=25 http://example.com/products/2/25

http://example.com/cgi-bin/feed.cgi?feed=news&frm=rss http://example.com/news.rss

http://example.com/services/index.jsp?category=legal&id=patents http://example.com/services/legal/patents

Don't reinvent a wheel

Relationships between URL and HTTP methods

Uniform Resource Locator (URL) GET POST

Collection, such ashttp://mgs.com/resources/

List the URIs and perhaps other details of the collection's

members.

Create a new entry in the collection. The new entry's URI is

assigned automatically and is usually returned by the operation.

Element, such ashttp://mgs.com/resources/item17

Retrieve a representation of the addressed member of the collection, expressed in an

appropriate Internet media type.

Not generally used. Treat the addressed member as a collection in its own right and create a new

entry within it.

Don't reinvent a wheel

HTTP API Design

Don't reinvent a wheel

Important HTTP Request HeadersHeader field

name Description Example

Cache-Control Used to specify directives that must be obeyed by all caching mechanisms along the request-response chain. Cache-Control: no-cache

Content-Type The MIME type of the body of the request (used with POST and PUT requests).

Content-Type: application/x-www-form-urlencodedContent-Type: application/json

Content-Type: multipart/form-data

Cookie An HTTP cookie previously sent by the server with Set-Cookie. Cookie: $Version=1; Skin=new;

HostThe domain name of the server (for virtual hosting), and

the TCP port number on which the server is listening. The port number may be omitted if the port is the standard

port for the service requested.Host: qa16-ats.mgspdtesting.com

Don't reinvent a wheel

Important HTTP Status CodesStatus Code Description

200 Ok

Standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request, the response will contain an entity describing or containing the result of the action.

302 Found Common way of performing URL redirection.

400 Bad Request The server cannot or will not process the request due to an apparent client error

403 Forbidden The request was a valid request, but the server is refusing to respond to it. The user might be logged in but does not have the necessary permissions for the resource.

404 Not Found

The request was a valid request, but the server is refusing to respond to it. The user might be logged in but does not have the necessary permissions for the resource. Usually means that the resource already didn’t exist.

501 Internal Server Error

Indicates that the request could not be processed because of conflict in the request, such as an edit conflict between multiple simultaneous updates.

Don't reinvent a wheel

Open Source Tools for API Testing• Performance testing tools• SoapUI (SoapUI NG Pro), Jmeter(SmartMeter, Blazemeter)

• Web Service testing tool• Postman, DHC, HTTPRequester, Wizdler, WebInject, Http Master, …

• Automation Frameworks• FitNesse, Cucumber, Robot Framework, …

Don't reinvent a wheel

SoapUI• Open-source web service testing application:• for service-oriented architectures (SOA) • for representational state transfers (REST).

• Covers:• web service inspection, invoking, development, simulation and mocking,• functional testing,• load and compliance testing.

• A commercial version, SoapUI Pro (Api Ready), which mainly focuses on features designed to enhance productivity.

Don't reinvent a wheel

SoapUI Open Source– pros and cons• Pros

• support HTTP(S), REST, SOAP, JDBC, AMF • IDE -Groovy -> command line, visible SoapUI API• User forums, tradition• Easy to learn, easy to use

• Cons• Team work support• No recording• Weak user documentation• Test history, test debugging, compare results• Buggy

Don't reinvent a wheel

Conclusions• Web application automated testing is possible through HTTP(S)• Web application automated testing is effective through REST• REST API popularity is growing

Don't reinvent a wheel

Something about fiddling

How to capture HTTPS communication

Don't reinvent a wheel

Fiddler• Captures HTTP and HTTPS traffic and logs it for the user to review

(the latter by implementing man-in-the-middle interception using self-signed certificates).• https://www.telerik.com/download/fiddler

Don't reinvent a wheel

HTTPS Traffic settings• Capture - F12 or Click on left bottom corner• Select captured process by• Preview - Tab Inspector• Request – Top part• Response – Bottom part

Don't reinvent a wheel

Icon help

Don't reinvent a wheel

Fiddler - Inspector• Request Inspectors• Headers—Shows request headers and status.• Webforms – Shows parameters visualized as table from QueryString and Body• Raw—Shows the “real” request header and body

• Response Inspectors• Headers—Shows response headers and status.• SyntaxView—Shows the response body with HTML syntax• WebView – Shows the browser output• Raw – Shows “real” response header and body

Don't reinvent a wheel

Fiddler - Settings

Tools\Telerik Fiddler Options\HTTPS Filters settings

Don't reinvent a wheel

Something about soap with UI

REST testing in SoapUI Open Source

Don't reinvent a wheel

Initial settings• File\Preferences – • Proxy Settings -> Manual 127.0.0.1:8888• UI Settings – Size of Req Response Message to Show – 1000000

• File\Import project (File\Create Empty Project)

Don't reinvent a wheel

Don't reinvent a wheel

What you get for 500$ in SoapUI NG Pro

• a• File\Import project (File\Create Empty Project)

Don't reinvent a wheel

Useful links

REST 101: The Beginner's Guide to Using and Testing RESTful APIs SoapUI - Getting Started with REST Testing SoapUI - Functional Testing SoapUI – Scripting & Properties SoapUI – API Docs

Don't reinvent a wheel

Reference List• Gojko Adzic, David Evans - Fifty Quick Ideas To Improve Your Tests• Mike Cohn - Succeeding with Agile: Software Development Using

Scrum• Lisa Crispin, Janet Gregory - Agile Testing: A Practical Guide for Testers

and Agile Teams• Dorothy Graham, Mark Fewster - Experiences of Test Automation:

Case Studies of Software Test Automation• Cem Kaner, James Bach, Bret Pettichord - Lessons Learned in Software

Testing: A Context-Driven Approach

Don't reinvent a wheel