formal testing of web content using ttcn-3 by robert probert, bernard stepien, pulei xiong...

Formal Testing of Web Content using TTCN-3

By Robert Probert, Bernard Stepien, Pulei Xiong

University of Ottawa

Motivation

• Web Applications (WA) are the most widespread target domain of software development in computer history.

• WA are error prone because mostly because anyone can develop them and they are mostly distributed.

• Testing of WAs: manual or using dedicated tools or robots.

Current state of researchon formal Web content testing

• XML based formalism

• Xiaoping Jia and Hongmin Liu, Rigorous and Automatic Testing of Web Applications

• Sahil Thaker, Robust Testing for Web Applications

Web content testing elements

• Hyperlink checking, including recursive• Text fragment checking• Protocol checking• Table checking• Form checking• Formatting checking• Mirror sites content checking• Version checking (languages,etc…)• Invalid HTML syntax checking• …

Specific needs of web content testing

• TTCN has been originally designed for telecommunication protocols.

• In Telecom, every single data element is relevant and needs to be coded and decoded to and from the abstract layer.

• In Web application, only part of the content is testable and needs to be extracted because most of the web page content is either too large or unpredictable.

• Thus, in web content testing, we test only invariants.

DePaul U Formalism BNFTestSpec ::= TestSuite*TestSuite ::= TestCase*TestCase ::= TestStep +TestStep ::= Name Condition opt RequestSpec

ResponseSpec TestStep*RequestSpec ::= URL RecursiveSpec opt

HearderSpec* ParameterSpec*VariableDecl*

ResponseSpec ::= StatusCode ContentTypeHearderSpec*VariableDecl* Predicate +

RecursiveSpec ::= Depth Domain optHeaderSpec ::= Name CDATA optional optParameterSpec ::= Name CDATA optional optCondition ::= PredicatePredicate ::= not Predicate | Predicate and predicate | Predicate or Predicate

| Predicate implies Predicate | ( forall VariableDecl + Predicate )| ( exists VariableDecl + Predicate ) | SimplePredicate

SimplePredicate ::= MatchPred | ContainPred | ComparePredMatchPred ::= SelectExp MatchOp ( RegExp | CDATA )MatchOp ::= equals | contains | startsWith | endsWithComparisonPred ::= Exp ComparisonOp ExpExp ::= NumberExp | DateExpComparisonOp ::= == | != | < | <= | > | >=ContainPred ::= HrefSpec | FormSpecHrefSpec ::= URL CDATA opt VariableDecl optFormSpec ::= ActionSpec opt MethodSpec optVariableDecl ::= Name SelectExp

bstepien

This BNF fits on a single slide. It is very compact and very targeted to specific web page content features.

Text Fragment checkingDePaul U formalism

<testcase name="Content check using predicates"> <teststep name="Content check step one"> <request url="http://www.cs.depaul.edu/program"/> <response statuscode="200"> <and>

<or> <match op="contains" regexp="false" select="/html/body"

value="Undergraduate Degree"/> <match op="contains" regexp="false" select="/html/body"

value="Bachelor Degree"/></or><match op="contains" regexp="true" select="/html/body”

value="[M|m]aster [D|d]egree"/> </and> </response> </teststep></testcase>

bstepien

The test case enables to perform a single test only. There is a concept of matching mechanism like in TTCN-3. However there are some interesting feature in this matching mechanism that are well adapted to web content testing needs like the logical expression on several values. Also, this matching mechanism enables some level of focussing on part of the data (select keyword).

Mapping DePaul U Formalism to TTCN-3

• Testsuite -> testsuite• Testcase -> testcase• Teststep -> function• Request -> send• Response -> receive• Match -> built-in TTCN-3 matching mech.• Regexp -> pattern• Value -> template

Text Fragment checking TTCN-3types and templates

template charstring program_page_web_url := "http://www.cs.depaul.edu/program";

type record content_response_type {charstring status,charstring content

}

template content_response_type content_response_pred := {status := "HTTP/1.1 200 OK",content := pattern "*<BODY

*[Undergraduate Degree | Bachelor Degree]*[[M|m]aster [D|d]egree]*</BODY>*"

}

bstepien

In TTCN-3, we first must define a record type that will handle both the returned status and the content value of a web page.The template will handle also handle both the status and the content.The logical expressions around the values are implemented using the logical operator of the TTCN-3 regular expression called the pattern.The meaning of the stars in TTCN-3 is: ignore any character until the specified string is matched.The TTCN-3 pattern feature handles both the DePaul U select and regexp features. For examplein the content pattern, the values Undergrate Degree must occur inside the BODY tags.

Text Fragment checking TTCN-3test case

testcase Content_check_pred_TC() runs on MTCType system SystemType {var verdicttype theOverallVerdict;

map(mtc:web_port, system:system_web_port);

theOverallVerdict := CheckContent_pred(program_page_web_url, content_response_pred);

setverdict(theOverallVerdict);}

bstepien

The TTCN-3 test case makes use of functions to maximize re-usability.

Text Fragment checking TTCN-3test logic

function CheckContent_pred(charstring theURLLink, content_response_type theContent_response) runs on MTCType return verdicttype {

var charstring theBadResponse;var verdicttype theFinalVerdict;

web_port.send(theURLLink);alt {

[] web_port.receive(theContent_response) {theFinalVerdict := pass;

}[] web_port.receive(charstring:?) -> value theBadResponse {

log("Bad Response: " & theBadResponse);theFinalVerdict := fail;

}}return theFinalVerdict;

}

bstepien

In TTCN-3 we can specify alternative responses to the request. The responses must match the content of the template (theContent_response).

Contains concept and TTCN-3 pattern

• The ‘and’ operator does not imply sequence.

• Permutations of values must be specified in the TTCN-3 template using pattern.

• Combinatorics explosion when the number of text fragments increases (np).

• TTCN-3 should implement a concept of “contains” with logical expressions

Evaluation of TTCN-3 advantages with text fragment checking

• DePaul U has the contains concept.

• TTCN-3 has better structuring of data sent or received.

• TTCN-3 has the concept of alternative to structure exceptions, etc…

• TTCN-3 is more flexible because new functionalities can be added both at the abstract and adapter level.

Recursive hyperlink checkingDePaul formalism

<testcase name="Link check"><teststep name="link check 1">

<request url="http://www.cs.depaul.edu", recursive="true", recursivedepth="3"/>

<response statuscode="200"/></teststep>

</testcase>

bstepien

This feature enables to perform web links integrity checking recursively. In order to avoid cycles there is a concept of maximum detph. It is unclear from their paper whther the response can be more than a status. For example a check to verify that all pages of a web site contain the company name could be of interest.

Recursive hyperlink checkingTTCN-3

testcase Link_Check_TC() runs on MTCType system SystemType {...theOverallVerdict := CheckChildLink(main_page_web_url, 1);setverdict(theOverallVerdict);

}

Solution: Test case invokes a recursive function and appropriate list type definitions.

type record of charstring HrefListType;type record hyperlink_response_type {

charstring status,HrefListType hrefList

}template hyperlink_response_type hyperlink_response :={

status := "HTTP/1.1 200 OK",hrefList := ?

}

bstepien

In TTCN-3, the recursion must be explicit. Consequently, the first step is to obtain the list of links on each web page. This list is composed of strings and the corresponding template uses the value passing feature "?" (any value).

Recursive hyperlink checkingTTCN-3 recursive function

function CheckChildLink(charstring theURLLink, integer theDepth) runs on MTCType return verdicttype {

web_port.send(theURLLink); web_port.receive(hyperlink_response) -> value theHyperlink_response;

if(theDepth <= maxDepth) {

var integer theNewDepth;var HrefListType theResponseHrefList :=theHyperlink_response.hrefList;var integer numOfLinks := sizeof(theResponseHrefList);var integer i;theNewDepth := theDepth + 1;for(i:=0; i < numOfLinks; i:=i+1){

oneVerdict := CheckChildLink(theResponseHrefList[i], theNewDepth);if(oneVerdict == fail) { theFinalVerdict := fail; }

} } return theFinalVerdict; }

bstepien

The recursion is implemented in a function since test cases can not be recursive. This function is invoked with two parameters, the current depth and the URL to check. The simple send/receive gets the list of links that is stored in a variable.Then, depending on the depth criteria we either return the current verdict (pass if the URL exists) or we go through the list of links and invoke recursively the funtion.

Evaluation of TTCN-3 advantages with recursive link checking

• TTCN-3 is more powerful because it allows more filtering on the recursion. For example, one would only recurse through the links to other web pages internal to the same company and ignore external links.

• TTCN-3 allows more sophisticated recursion control than simple depth limits.

• DePaul formalism is more compact but also more limited.

Quantified predicate checkingDePaul U Formalism

<testsuite> <testcase name="Content check using quantified predicates"> <teststep name="content check step one" > <request url="http://jordan.cs.depaul.edu/webtest/testhref.htm"> </request> <response>

<forall><varbale name="l" select="descendent::a/@href"/><match op="startswith" select="$l" regexp="false"

value="http://"/></forall>

</response> </teststep> </testcase></testsuite>

bstepien

This construct allows to verify if all href link uses a specific protocol like here http://.The forall keyword ensures that all links are checked.

Mapping to TTCN-3

• Forall -> TTCN-3 for loop with bounds

• StartsWith -> TTCN-3 substring

• Select -> done in TTCN-3 CODEC

bstepien

while most of these features can be implement at the TTCN-3 abstract level, the select feature can only be implemented at the adapter level in the codec.

Quantified predicate checkingTTCN-3

testcase Forall_protocol_check_TC(…) runs on MTCType system SystemType {

web_port.send(theURLLink); web_port.receive(hyperlink_response) -> value theHyperlink_response;

all_match := true; for(i:=0; i < numOfLinks; i:=i+1) {

var HrefListType theResponseHrefList := theHyperlink_response.hrefList; if(substr(theResponseHrefList[i],0,theLength) != theProtocol) {

all_match := false;log("link: " & theResponseHrefList[i] &

" is not of protocol " & theProtocol);}

} if(all_match) { setverdict(pass); } else { setverdict(fail); } }

bstepien

The TTCN-3 test case consist mainly in obtaining the list of links and processing that list to verify that each link starts with the appropriate protocol.The important point here is that we can not use the template matching mechanism to perform this function but rather a classic sub string comparaison like in any programming language.

Remarks about the TTCN-3 solution

• Matching is not achieved via the inherent TTCN-3 matching mechanism, but instead using traditional programming languages features (substr). This is mainly because it is impossible to match a list of unknown length.

• There is no parametric solution. Two different select values would require a dedicated definition of types, templates and test cases for each select values.

Our Recommendationparametric CoDec

type record myRecord {integer field_1,charstring field_2 codecparameter, charstring field_3

}

template myRecord myTemplate := {field_1 := 5, field_2 := “href”, field_3 := “abcd”

}

Some fields act as parameters passed to the CODEC rather than matching values coming from the CODEC

bstepien

field_2 value "href" is a parameter to the codec and should not be used in the matching mechanism.

Sahil Thaker WebDeveloppers ltdForm testing formalism

<oracle scope = form><form method=“get” target=“”><input type=hidden name=“” value=“”><input type=submit>

</oracle>

bstepien

This is a feature that allows to check names and values of form elements.

Form testing in TTCN-3type definitionstype record FormResponseType {

charstring status,FormType form

}type record FormType {

charstring name,charstring method,charstring action,InputListType inputList

}type record InputType {

charstring name,charstring type_input,integer size_input,charstring value_input

}type record of InputType InputListType ;

bstepien

In TTCN-3 we must first define the appropriate data types to describe a form. Here we use a list to describe the form input elements.

Form testing in TTCN-3template definitions

template FormType theRegistrationForm_template := { name := "subscription_form", method := "POST", action := "http://www.someone.com/cgi-bin/processSubscription.pl", inputList := { {name :="firstname", type_input := "text", size_input := 20, value_input := ""} , {name :="lastname", type_input := "text", size_input := 35, value_input := ""} , {name :="city", type_input := "text", size_input := 40, value_input := ""}, {name :="send_me_emails", type_input := "checkbox",

size_input := 0, value_input := ""} , {name :="wine_and_cheese", type_input := "checkbox",

size_input := 0, value_input := ""} , {name :="", type_input := "submit", size_input := 0, value_input := "send"} , {name :="", type_input := "reset", size_input := 0,

value_input := "Oooops!_Let_me_try_again!"} }}

bstepien

A complete form can be described in a template. The interesting thing is that template can be derived at design time and thus enable if the implementation is compliant to the design, providing the implementor uses the same names as the designer.

Form testing in TTCN-3parametric testcase definition

testcase Form_Check_TC(charstring theURLLink, FormResponseType theFormResponse) runs on MTCType system SystemType {

…

web_port.send(theURLLink);alt {

[] web_port.receive(theFormResponse) -> value theFormResponseTypeValue{

log("the form " & theFormResponseTypeValue.form.name & " conforms to specification");

setverdict(pass);}[] web_port.receive(FormResponseType:?) {

log("in the catch all receive !");setverdict(fail);

}}

}

bstepien

This illustrates the benefits of TTCN-3 parametric test cases. A generic test case can be used to test several different forms.

Form testing in TTCN-3test control definition

control {execute (Form_Check_TC(form_page_web_url,

form_response(theRegistrationForm_template)));execute (Form_Check_TC(another_web_page_url,

form_response(another_form_template)))}

Maximizing re-usability using:•Parametric test cases•Parametric data templates

bstepien

This illustrates that the parametrization can be performed at two levels, test cases and template values.

Advantages of TTCN-3

• Parametric test cases

• Parametric templates

• Structuring at all levels, types, templates, test cases, functions

• Concept of control

• Test configuration specification

Conclusions

• TTCN-3 is adequate for web content testing.

• There is no need to create new formalism dedicated to web content testing.

• Instead, it would be more beneficial to use TTCN-3 to develop generic web content testing tools mostly because of TTCN-3 parametric features.

formal testing of web content using ttcn-3 by robert probert, bernard stepien, pulei xiong...

Documents