black-box testing technique for information visualization

Black-Box Testing Technique for InformationVisualization. Sequencing Constraints with

Low-Level InteractionsMartın L. Larrea1

1Laboratorio de Investigacion y Desarrollo en Visualizacion y Computacion Grafica, VyGLab, Departamento deCiencias e Ingenierıa de la Computacion, Universidad Nacional del Sur, San Andres 800, Bahıa Blanca, Argentina,

[email protected]

AbstractThe software development process has matured significantly o ver t he p ast d ecade. W e are currently in a state where the need for the verification and validation o f the product under development is unquestionable. Visualizations, as software products, must go through such verification a nd v alidation c ontrol. A t the implementation level, a visualization software is no different from any other software, its source code can be verified a nd v alidated by using any available white-box technique. The usability studies and controlled experiments are helpful to understand how the user perceives and uses the visualization. However, at an interaction level, a visualization software is not like any other software. Most interactions on a traditional software, are based on buttons and text fields w hile o n a v isualization, t he interactions are mostly based on zooming, selecting and filtering visual e lements. The black-box techniques found in the literature, also known as functional tests, are not suitable for this context. This paper describes a black-box technique tailored for information visualization implementations. The technique is built on constraints imposed over the sequences of low-level interactions available in the visualization and the User Action Notation.Keywords: V isualization systems and software,Software/Program Verification.

1 IntroductionVerification and Validation (V&V) is the processof checking that a software system meets itsspecifications and fulfills its intended purpose.The software engineering community have ac-knowledged the importance of V&V process toensure the quality of its software products. Inits essence, the process consists of selectingelements from the program, module or functioninput domain, executing the program with thoseelements as input and comparing the actualoutputs with the expected outputs. Even though

the importance of the V&V process is wellknown and documented, it has commanded apartial attention in the information visualizationcommunity.

A search on IEEE Xplorer, ACM Digital Libraryand Google Scholar, with the query “visualizationtesting verification validation” returns a list ofpapers where the focus is not on the functionaltesting of the visualization software. Thesepapers focus on the problem of evaluatingvisualization system for effectiveness on userexperience. Without detracting from usabilitytesting, functional testing is crucial in any softwaredevelopment. All visualization software mustbe submitted to the V&V process, pointed outby Kirby and Silva in [1]. On the other hand,Graphical User Interfaces (GUI) Testing is a morepopular and developed topic. GUI Testing is asystem testing of a software that has a GUI front-end. There are many research papers on thistopic ([2, 3, 4, 5, 6]). As discussed later, theseworks are not well suited to the characteristicsof a visualization. We will focus in one GUItesting technique in particularly, the User ActionNotation (UAN), as it is one of the basis of ourproposal.

The V&V process, also known as softwaretesting or just testing, is composed of V&Vtechniques. There are many different V&Vtechniques which are applicable at different stagesof the development lifecycle. The two maincategories of testing techniques are white-box andblack-box. In the first one, the testing is driven bythe knowledge and information provided by theimplementation or source code. In the secondone the specification of the software, moduleor function is used to test the object underreview. Black-box testing is not usability testing;Usability testing tests the extent to which a usercan learn to operate, prepare inputs for, andinterpret outputs of a system or component. Black-box testing evaluates high-level design and thecustomer requirements specification, to ensure thesystem does what it is intended to do.

All black-box testing techniques, design its testscases based on the software specification. Some

Recceived 11 October 2017 / Revised 11 December 2017 / Accepted 21 February 2017

JCS&T Vol. 17 No. 1 April 2017

37

of these techniques involve the GUI componentsof the software and the interactions with them.Buttons, text fields and drop down list arecommon elements among those GUI. Nonetheless,a visualization constitutes a GUI by itself withmore components than a regular user interface.Beside buttons and text fields, a visualization mayhave glyphs, 3D or 2D visual objects that changelocation or shape according to users interactions.In these cases, the black-box techniques that relyonly on common GUI components are not suitable.

Those techniques which do not involve graphiccomponents, use decision tables ([7]) or otherforms of tabular representation to test the software.Among those, there are very informal techniqueswhich are very difficult to methodize and relyheavily on the tester goodwill. Others allowsystematizing the testing by using a formalspecification, which is very complicated to achievefor a visualization ([8]).

A visualization software, at the source codelevel, is just another piece of software and assuch, can be tested with any available white-box technique ([9]). Furthermore, researchershave investigated whether and how visualizationsoftware features can improve the effectiveness anduser perception of information systems. However,there are still challenges ahead, in particularly, theblack-box testing of information visualizations isan uncharted area. If the visualization communitywants to embrace the V&V process as a wayto ensure the quality of their software products,then we need to adapt existing techniques to thiscontext or to define new ones. To respond to thisgap, in this work a black-box technique based onUAN and sequences of low-level interactions isintroduced.

We can identify two levels of interactions onvisualization, high and low-levels. High-levelinteractions describe the reasons for the creationof the visualization, the user goals. Differentauthors describe different high-level interactions;Kovalerchuk and Schwing established ([10]) threehigh-level interactions: Illustrate, Reason andDiscover ; while Keller and Keller ([11]) describednine. Low-level interactions are those providedby the visualization to achieve the high ones i.e.the user’s goal. Shneiderman ([12]) identifiedseven low-level interactions: Overview, Zoom,Filter, Detail on demand, Relation, History andExtraction.

Low-level interactions exist at the implemen-tation level of the visualization, while high-levelinteractions are more abstract. Because we wantto design a testing technique for informationvisualization implementation, this article focuseson low-level interactions.

In the remaining parts of this article, we first

review the concepts of V&V in the softwaredevelopment process as well as in the contextof visualizations. The following section providesbackground V&V in general and applied toGUI and visualizations. Then, we describe theproposed technique, a black-box testing methodsuitable for information visualizations. Weintroduce 2 examples of the use of such techniqueon existing visualizations. We conclude with abrief discussion on limitations and advantages ofour approach and the proposed future work.

2 Background Review

2.1 Background on Verification & Val-idation

Software testing is involved in each stage ofsoftware life cycle, but how we test and whatwe test on each stage of software developmentis different, the nature and the goals of what isbeing tested are different. Based on [13], thereare 8 types of testing in the life cycle: Unittesting is a code based testing which is performedby developers, this testing is mainly done totest each and individual units separately. TheUnit testing can be done for small units of code,generally no larger than a class. Integrationtesting validates that two or more units worktogether properly, and inclines to focus on theinterfaces specified in low-level design. Systemtesting reveals that the system works end-to-end in a production-like location to providethe business functions specified in the high-leveldesign. Acceptance testing is conducted bybusiness owners; the purpose of acceptance testingis to test whether the system does, in fact,meet their business requirements. RegressionTesting is the testing of software after changeshave been made to ensure that those changesdid not introduce any new errors into thesystem. Alpha Testing is usually applied at thedeveloper’s site, with the presence of the developer.Beta Testing is done at the customer’s site withno developer in site. Functional Testing is donefor a finished application, its goal is to verify thatthe system provides all the required behavior.

In the context of V&V, black-box testing is oftenused for validation (i.e. are we building the rightsoftware?) and white-box testing is often usedfor verification (i.e. are we building the softwareright?). In black-box testing, the test cases arebased on the information from the specification.The software testers do not consider the internalsource code of the test object. The focus of thesetests is solely on the outputs generated in responseto selected inputs and execution conditions. Thesoftware tester sees the software as a black box,where information is input to the box, and the box


38

sends something back out. This can be done purelybased on the requirement specification knowledge;the tester knows what to expect from the blackbox to send out and tests to make sure that thebox sends out what it’s suppose to send out.

Oppositely, white-box testing, also calledstructural testing, designs test cases based on theinformation derived from the source code. White-box testing is concern with the internal mechanismof a system, it mainly focuses on the control flowor data flow of the program. White-box andblack-box testing are considered to complementeach other. In order to test software correctly,it is essential to generate test cases from thespecification and source code. This means thatwe must use white-box and black-box techniqueson the software under development.

Each test technique, both white-box and black-box, must describe a test model and, at least, onecoverage criteria. Test models describe how togenerate test cases, it can be a graph, a table or aset of numbers. Coverage criteria are used to steerand stop the test generation process ([14]), theyare usually boolean conditions. They have widelyaccepted means of assessing the quality of a test([15]). Both concepts will be revisited later, wherewe will discuss a particular testing technique.

2.2 Background on Verification & Val-idation in Visualization

In 2008, Kirby and Silva ([1]) stated the need toincorporate the V&V process into the visualizationprocess. They said: To gauge the extent to whichverification already exists in the visualizationcommunity, we reviewed papers from the past fiveyears (2003 to 2007) of the IEEE Visualizationconference (http://vis.computer.org). To establishthe extent to which the nomenclature is alreadyused, we searched the texts of these papers foroccurrences of “verify”, “validate” and otherforms of these words. Fewer than half of thepapers ever used any of these words, let alonein the context of validation or verification. Tocheck whether these concepts exist, we revieweda subset of the papers in greater detail. Whilemany include sections presenting a logical andmathematically sound derivation of their techniqueand evaluation of their algorithms’ performance,only a handful discussed concepts related to eithervalidation or verification. Those that did tendnot to adhere to any standard organization orsystematic methodology.

We extended the search Kirby and Silva didand included ACM Digital Library, IEEE Xplorerand Google Scholar up to 2015. We foundequivalent results; Xiaojun et. al ([16]) testhow users perceive visualizations, Koshman ([17])talks about testing user interactions but he

actually does a test on user’s cognitive preattentiveprocessing. Aerts et. al ([18]) test the effectivenessof two visualization techniques. These are someof the works with titles that mention “testing”,“verification” or “validation” and when readingthem, they were actually some sort of usabilityevaluation. There were many more than explicitlymention usability testing in their title.

Etiene’s work ([19, 20, 21]) on verifiablevisualization checks the mathematical calculationsinvolved in the visualization process. He statedthat visualization, scientific volume renderingspecifically, has not fallen under the same rigorousscrutiny as other components of the pipeline likemathematical modeling and numerical simulation.These works are more related to a white-boxtesting technique approach.

There is a significant effort into usabilityand user experience testing in the contextof information visualization but no equivalentendeavor into black-box techniques. If asoftware is only tested by white-box and usabilitytechniques then this means that we are not doingour best to ensure the quality of the final product.

2.3 Background on Verification & Val-idation in GUI

Banerjee et al. ([2]) define the term GUI testingto mean that a GUI based application, i.e., onethat has a GUI front-end, is tested solely byperforming sequences of events (e.g., “click onbutton”, “enter text”, “open menu”) on GUIwidgets (e.g., “button”, “text-field”, “pull-downmenu”). From the user’s point of view, GUIs offermany degrees of usage freedom, i.e., users maychoose to perform a given task by inputting GUIevents in many different ways in terms of theirtype, number and execution order.

Banerjee et al. also provided a study ofthe existing body of knowledge ([2]) on GUItesting since 1991. Hellman et al. ([22])presented a review of test driven developmentof GUI and Memon and Nguyen ([23]) presenteda classification based on model-based GUI testtechniques.

Hellman et al. stated that GUI testing is verydifficult in part due to the degree of freedom GUIsallow users. GUIs can enter a large number ofpossible states in response to user input, and it isoften difficult to determine the validity of a givenstate in an automated fashion. This becomesmore complex when we consider an informationvisualization. Among the techniques reviewed byHellman et al. we can found Capture Replay Tools(CRT), Invariant Based Testing and Model BasedTesting.

CRT techniques are based on recording themouse and keyboard activities as scripts in order


39

to replay them later. These techniques are notsuitable for information visualization becausevisual elements are not always in the same place.The same visualization technique on two differentdata set will produce different representations. Arecord from one representation will not work onthe other.

In Invariant Based Testing rules which defineexpected or prohibited system behavior arecreated. After each interaction with the GUIelements, the invariants are checked. Thisapproach is based on knowledge of how the GUIis implemented, so this is a white-box technique,not the subject of our study.

Finally, Model-based testing requires thegeneration of an intermediate version of the GUI,either by reverse-engineering a working GUI orgenerating one based on specifications. The maindisadvantage of this type of techniques is that amodel must be created and maintained. If themodel describes the GUI widgets, which vary alot during the development of an interface, thenthe maintenance of the model will require a lot ofeffort.

Memon and Nguyen ([23]) acknowledgedthat quality assurance is becoming increasinglyimportant for GUIs as their functional correctnessmay affect the quality of the entire system in whichthe GUI operates. They presented 16 model basedtesting techniques. These techniques require thecreation of a model, like a table, a graph or a treeand an algorithm to create test cases based onthe model. As we said earlier, all models thatare based on GUI widget must be updated everytime there is a modification in the interface. Theadvantage of model based testing is that, havinga model with a formal representation allows forthe use of an algorithm to create the test cases.In many cases, this algorithm can be automated.

Banerjee et al. ([2]) developed a classificationscheme for 136 articles about GUI testing.They concluded that there is a large focuson model based testing in the literature, butnone of the commercially available tools aremodel based. Most commercial tools are basedon CRT techniques. They saw this as asignal of disconnection between researchers andpractitioners.

3 Our Proposal

Our goal in this work is to define a testingtechnique suitable for information visualization,focusing on a black-box approach. We want toprovide an easy to use methodology, with aneasy to read and understand representation ofthe visualization’s interactions. More particularly,we want a method that the developer or even

the user of the visualization can use, withoutthe need of a testing specialist. We want toreduce the disconnection between commercialtools and literature proposal, between researchersand practitioners, mentioned by Banerjee et al([2]).

Our proposal is based on two previous work,User Action Notation and Sequencing Constraints.Both concept are introduce in the followingsubsections and then our technique for black-boxtesting on Information Visualization is describe.

3.1 User Action NotationUser Action Notation (UAN) is a task and user-oriented notation for behavioral representationof asynchronous, direct manipulation interfacedesigns ([4, 6]). At the lower level, user actions areassociated with feedback and system state changes.The notation makes use of visually onomatopoeicsymbols. UAN was created as a tool-supportedtechnique capable of specifying the behavioralaspects of interaction systems, the tasks and theactions a user performs to accomplish those tasks.

The example of UAN presented by Hartson etal. ([4]) can be used here to clarify how it is used.Consider the task moving a file icon, which canbe described in prose as:

1. Move the cursor to the file icon. Depress andhold down the mouse button. Depressing themouse button selects the file, indicated bythe highlighting of its icon.

2. With the button held down, move the cursor.An outline of the icon follows the cursor asyou move it around.

3. Release the mouse button. The display of theicon is now moved to where you released thebutton.

These actions are described in UAN as:

1. ∼ [file icon]M∨

2. ∼ [x, y]∗ ∼ [xÍ, yÍ]

3. M∧

The ∼ denotes moving the cursor, in this caseinto the context of the file icon. M∨ representsdepressing the mouse button. In the second part∼ [x, y] indicates movement of the cursor to anarbitrary point x, y on the screen. The ∗ meansto perform, zero or more times, the task to whichit is attached. Thus, ∼ [x, y]∗ ∼ [xÍ, yÍ] means tomove the cursor to a succession of zero or morearbitrary points about the screen, ending at thepoint xÍ, yÍ. Finally, in the third, part the mousebutton is released.


40

UAN primary goal is the representation of thebehavior of the interactions and the system state,in the interface development process. It wasnot intended to use as a testing tool. It has arich symbolic alphabet and even allows for therepresentation of the system state and its changes.Its power of expression means that some simpleactions described in UAN can be difficult to readand interpret.

3.2 Sequencing Constraints

In 1994, Kirani and Tsai ([24]) presented atechnique called Message Sequence Specificationthat, in the context of an object oriented program,describes the correct order in which the methodsof a class should be invoked by its clients. Themethod sequence specification associated with anobject specifies all sequences of messages that theobject can receive while still providing correctbehavior.

Their strategy uses regular expressions tomodel the constraints over the correct order ofthe invocation of the methods i.e. the regularexpression is the test model. Method nameswere used as the alphabet of a grammar whichwas then used to statically verify the program’simplementation for improper method sequences.A runtime verification system identifies incorrectmethod invocations by checking for sequenceconsistency with respect to the sequencingconstraints.

If a class C has a method M1, this is notedas CM1 . Sequence relationships between twomethods were classified into three categories,sequential, optional and repeated. If the methodM1 of C should be invoked before the methodM2 of the same class, then this relationship issequential and is represented as

CM1•CM2

If one, and only one of the methods M1 and M2can be invoked then this relationship is optionaland is represented as

CM1 |CM2

Finally, if the method M1 can be invoked manytimes in a row then this is a repeated relationshipand is represented as

(CM1)∗

For example, if a class X has three methodscreate, process and close, a possible sequencingconstraint based on Message Sequence Specifica-tion could look like

Xcreate•(Xprocess)∗•Xclose

If class X is part of a larger system S, thenwe could statically check the source code of S tosee if all calls to X’s methods follow the definedgrammar. If a static analysis is not enough, wecould implement a runtime verification systemthat tracks all calls to X’s methods and checks

dynamically the sequence of calls against thegrammar.

This technique can also be used to test therobustness of a system. Continuing with theclass X as an example, we can use the definedgrammar to create method sequences that arenot a derivation from the grammar, i.e. incorrectsequences methods. These new sequences can beused to test how the class handles a misuse. Forexample, how does the class X respond to thefollowing sequence of calls:

Xcreate•Xclose•Xprocess

Testing with the sequences generated by thegrammar and with those that were not, are the twotechniques presented in [26] a work that extendedthe research done in [24].

3.3 Testing Information Visualizationusing Sequencing Constraints withLow-Level Interactions

UAN is a suitable representation of the interac-tions in a GUI and, in our case of interest, in avisualization. But it has a complex alphabet andits output is not easy to read. Kirani and Tsai([24]) share multiple similarities with UAN, whereinstead of working with GUI interactions theyused class methods. Their work does not hold thesame expressivity that UAN but it is clearer andeasy to interpret.

In this work, we propose a merge of them inorder to design a black-box testing technique forInformation Visualization. Ji Soo et al. ([25])described a visualization system as a two-partsystem, representation and interaction. They said:“The representation component, whose roots liein the field of computer graphics, concerns themapping from data to representation and how thatrepresentation is rendered on the display. Theinteraction component involves the dialog betweenthe user and the system as the user explores thedata set to uncover insights.”

The user uses the interactions to express ordersto the visualization, the interactions work as alanguage and as such they may have restrictionsor rules on how is used. Kirani and Tsai ([24])idea of identifying the restrictions for the correctorder in which the methods of a class could beinvoked can be applied here. Our proposal is touse the basis of what was originally design as awhite-box testing technique for object orientedprograms in order to develop a black-box testingtechnique for visualization implementations.

3.4 Our Contribution

Instead of writing a sequence constraint on themethods of a class, we will write a SequenceConstraint on the Interactions (SCI). Each SCI


41

involves a set of binary or unary operators anda set of symbols. For simplicity, for now onthe symbols will represent the actual interactionsavailable in the visualization. Following the workby Kirani and Tsai ([24]), the operators will be:Sequential: If an interaction I2 must always goafter the interaction I1 then there is a sequentialrelationship among them, denoted as I1•I2.Optional: If the user can choose between twointeractions, said I2 and I1, then there is anoptional relationship among them, denoted asI1|I2. Note that in this case, the notation I2|I1 isequivalent.Repetition: If the user can use the interactionI1 multiple times in a row, then it is a repetition.Unlike the work done at [24] and [26], we introducetwo types of repetition, one that implies that atleast one time the user must use I1 and the otherthat allows for zero appearance of I1. The symbol∗ will be used to represent cardinality 0 or more,and the symbol + will be used for 1 or more. IfI1 can be used zero or more times, then this isrepresented as I1

∗; if I1 must be used at least onetime, it is expressed as I1

+. I1+ is equivalent to

I1•I1∗.

These elements can be combined to formmore complex expressions. If the user can useone of three interactions multiple times, thiscan be expressed as (I1|I2|I3)+. In this caseby using the symbol + we are saying thatat least one of the interactions must be usedonce. Repetition operators have precedence overSequential and Optional operators. The Optionaloperator takes precedence over the Sequentialone. Parentheses can be used to define theinterpretation of a SCI. Suppose we have threeinteractions I1, I2 and I3, then the following SCI

I1+•I2|I3

expresses that first we must consider theSequential operator, use I1 one or moretimes and then we must choose betweenusing I2 or I3. By using parentheses wecan change the interpretation of the SCI

(I1+•I2)|I3

In this case, we first consider the Optionaloperator, we must choose between using I3 orthe expression between the parentheses.

We can use the symbols in the SCI to representmore than the available interactions, we can alsouse them to modularize expressions. Let usimagine a visualization VX with six interactions,Open, Pan, Zoom, Selection, Detail and Close.Open represents the creation of the visualization,from opening the source data to setting thevisualization process; when Open concludes theuser has the actual visualization on screen. Detailrepresents detail on demand and can only be usedif the user selected something using Selection.

Pan and Zoom allow the user to explore thevisualization. The following grammar representsthe constraints over the sequencing of interactionin VX , considering O for Open, P for Pan, Z forZoom, S for Selection, D for Detail and finally, Cfor Close.

SCI for VX : O•(O|Z|P |(S+•D∗))∗•CWe can define two new symbols Selection&Detailand MainInteraction, and redefine the SCI for VX :

Selection&Detail = S+•D∗

MainInteraction = O|Z|P |Selection&DetailSCIforVX : O •MainInteraction •C

This grammar means that the first validinteraction with VX is Open, then the usercan Open again or Zoom or Pan or Selection.Note that if the user want Detail, first theuser must do at least one Selection. Theinteraction with VX ends when the user endsthe visualization with the Close interaction. Thesymbols Selection&Detail and MainInteractionallow a clearer expression for such behavior.

3.5 Coverage Criteria

In V&V the term Coverage Criteria is a measureor indicator of the amount of test to perform. Inother words, it answers the question “how muchtest should we do?”. Each testing technique, blackor white-box, have its own indicator of CoverageCriteria. A single technique may have manyCoverage Criteria and among them there is aquality/complexity relationship. On one hand, wefind very simple criteria which will generate simpletest cases with a not too extensive coverage; whileon the other end, there will be complex criteriathat will generate very complex test cases. Theformer detects fewer problems than the later.

With our proposal of Sequencing Constraintswith Low-Level Interactions there are two typesof test cases that we can generate, valid test casesbased on valid interaction sequences and invalidtest cases based on interaction sequences thatcannot be derived from the defined grammar.

Let I be the set of interactions available on thevisualization VX , and G, the SCI for VX using theelements of I. Consider T to be the set of test caseswhere each case is a sequence of the interactionsin I. With these elements, we can now introducethe Coverage Criteria for Sequencing Constraintswith Low-Level Interactions. These criteria aredivided into two categories, as in [26], coveragecriteria for valid sequences and for invalid ones.The criteria on each category were created for ourtechnique.


42

3.6 Coverage Criteria for Valid Se-quences

Base Coverage: Let i is the minimum length ofvalid sequences derived from G, then T satisfiesthe Base Coverage Criteria if and only if Tcontains all the possible sequences derived fromG of length i. If i equals 0 then T is an empty setand satisfies the Base Coverage Criteria.

Base+1 Coverage: Let i be the minimumlength of valid sequences derived from G, then Tsatisfies the Base+1 Coverage Criteria if and onlyif T contains all the possible sequences derivedfrom G of length i + 1.

Base+n Coverage: This is a generalizationof the previous coverage. Let i be the minimumlength of valid sequences derived from G, then Tsatisfies the Base Coverage Criteria if and only ifT contains all the possible sequences derived fromG of length i + n, where n >= 2. It is importantto note that G may impose limits on how large ncan be.

3.7 Coverage Criteria for Invalid Se-quences

Invalid Coverage: T satisfies the InvalidCoverage Criteria if and only if T contains allthe possible sequences of length 1 that are notderived from G.

Invalid-2 Coverage: T satisfies the Invalid-2Coverage Criteria if and only if T contains all thepossible sequences obtained from the combinationof 2 interactions of I but are not derived from G.

Invalid-n Coverage: T satisfies the Invalid-nCoverage Criteria if and only if T contains all thepossible sequences obtained from the combinationof n interactions on I, where n >= 2, but are notderived from G.

3.8 Examples of Coverage Criteria

Continuing with the SCI G defined for VX ,G = O•(O|Z |P |(S+•D∗))∗•C ,

we can now generate test cases basedon the different criteria. Let T1be a set of test cases as follows:

T1 = {O•C}T1 does satisfy the Base Coverage Criteria

because G does not allow an empty sequence or asequence of length 1, and Open must always bepresent at the beginning and Close always endsthe sequences. Because the middle of the SCI isenclosed in ∗ we can consider it empty. So, thesmallest length for valid sequences from G is 2.Then can expand T1 and create T2 as:T2 = {O•C, O•O•C, O•Z•C, O•P•C, O•S•C}T2 satisfies the Base+1 Coverage Criteria

because it contains all the possible sequences

Figure 1: Theme river visualization for themigration of people within the United States

derive from G of length 3, which is +1 on thesmallest length possible. T2 also contains allthe possible sequences from G of length 2, soit also satisfies Base Coverage Criteria. If we nowconsider T3:T3 = {O•C, O•Z•C, O•P•C, O•S•C, O•S•S•D}

T3 satisfies Base Coverage Criteria but doesnot satisfy Base+1 Coverage Criteria because itdoes not contain the (O•O•C) sequence, it neithersatisfies Base+2 Coverage Criteria as it only hasthe sequence (O•S•S•D) of length 4.

These Criteria are based on the length of thesequences of interactions. Some SCI will imposean upper limit on that value but others may not.When there is no limit on how long these sequencescan be, is up to the person using this testingtechnique to decide how far to go.

4 Test Cases using SequencingConstraints with Low-Level In-teractions

In this section we will use our technique togenerate test cases on two visualizations. Thechosen visualizations were part of the BestVisualization Projects of 2014 by the Flowingdatawebsite1.

4.1 Where We Came From and WhereWe Went

This visualization2 shows how people in the UnitedStates have moved between states since 1900. AsFig. 1 shows, the authors of the visualizationused a theme river technique ([27]) for the visualrepresentation of information. The site offers 51visualizations, one for each state. The jump fromone visualization to another is achieved through aselector on the top of the web page. The low-levelinteractions are:

• Select Origin State: This is done by theselector on the top. After choosing a state,

1http://flowingdata.com2http://www.nytimes.com/interactive/2014/08/13/upshot/where-

people-in-each-state-were-born.html


43

the visualization will show in percentageformat, where the people born in that statehave move to.

• Switch to Out of State: This interactionchanges the visualization to show where thosewho live in the selected state are from. It isdone by clicking a button.

• Detail: When the user moves the mouse overthe different rivers in the visualization, a popup box appears showing the exact percentvalue at a specific time for a specific location.This interaction is possible in the Origin Stateand in the Out of State visualizations.

To do Sequencing Constraints with Low-LevelInteractions we will replace each interaction witha simplified representation, Select Origin Statewill be SO, Switch to Out of State will be LOand Detail will be De. The visualization, as itis presented in the website, start automaticallywith where people born in California have moveto. There is no need to explicitly load the firstdata set. There is no restriction in the order onwhich of the three low-level interactions can beused, this mean that the SCI for this visualizationis

(SO|LO|De)∗

4.1.1 Coverage Criteria for Valid Se-quences

Because no interaction in this example isobligatory, the minimum sequence of interactionvalid for this visualization is the empty sequence.Because of this, the Base Coverage is achieved byjust running the visualization, i.e. without anyinteraction. We access the website, and thus thevisualization appears with California as defaultstate. So the visualization behave correctly.

In order to achieve Base 1 Coverage we needto provide a set of test T such as T contains allthe possible sequences of length 1 derived from(SO|LO|De)∗. T1 satisfies this condition.

T1 = {SO, LO, De}T1 means that we must execute the visualization

three times. The first time we change the originstate and only do that, for example we choseto change from California to Texas. On thesecond run, we switch to Out of State. Becausethe default state is California, we change thevisualization from “Where the people born inCalifornia have move to”, to “Where are fromthose who live in California”. On the final runfor T1 we use the interaction Detail. On thevisualization for California we position the mouseon the vertical line corresponding to the year 1950,in the Stayed in California river. In all three cases

the visualization behaves as expected, so we canconclude that T1 was passed.

The following test set, T2 accomplish Base 2Coverage and was also passed by the visualizationsuccessfully.T2 = {SO•SO, SO•LO, SO•De, LO•S, LO•LO,LO•De, De•SO, De •LO, De•De}

4.1.2 Coverage Criteria for Invalid Se-quences

Because the sequence constraints for this visu-alization is (SO|LO|De)∗ there are no invalidsequences. Hence, it is not possible to satisfyany Invalid Coverage.

4.2 How the Recession Reshaped theEconomy

This visualization3 shows how during the ten yearsfollowing the great recession experienced in theUnited States, 9 million lost jobs were recovered.

For the visual representation of information,the authors chose a line graph, where each linerepresents a different industry. The X-axis is usedfor the level of wages, from lower to higher; andthe Y-axis is used from the number of jobs gainor lost since the recession. Each line shows a timeperiod from 2004 to 2014.

This visualization is far more complex than theprevious one, in its content and interaction model.We can divide the visualization into three parts,where the set of available low-level interactionon each part is different: the initial visualization(Figure 2), the main one (Figure 3) and finally thedetailed representation (Figure 5).

The initial visualization (Figure 2) is the onethat appears when the user opens the visualizationweb page. This a compact representation of themain visualization. The Y-Axis is compressed inorder to gain space for the title and description.Like the previous example, the data set is loadedautomatically when the user enters the website.We can detect four low-level interaction in thisstage.

• Hover over line: This interaction works asDetailOnDemand. When the user placesthe mouse on any line in the visualization,a small visualization appears showing moredetail about the industry in display. We willuse the symbol Hline for this.

• Go through the detail: When the user movesthe mouse over the small visualization createdby Hline, the information in it is updatedbased on the position of the mouse within.We will use the symbol D for this. Because

3http://www.nytimes.com/interactive/2014/06/05/upshot/how-the-recession-reshaped-the-economy-in-255-charts.html


44

Figure 2: Visualization presented to the userwhen he/she enters the website. It is a compactrepresentation of the main visualization.

Figure 3: The main visualization of how jobs werecreated or lost during a ten year period on differentindustries.

D must always go after Hline we will createa new symbol Z = Hline•D∗. Z indicatesthat the user can go through details afterplacing the mouse over a line, and he/she cango through details many times.

• Click Scroll & Mouse Scroll: These are twointeraction that allow the user to move tothe next part of the visualization. On thebottom of the initial visualization there is alabel titled “Scroll” that when the user clickson it, changes to the main visualization. Thesame effect is achieved by using the mousescroll. We will use Cscroll and Mscroll forthese ones.

The SCI for the initial visualization, calledVinitial is

Vinitial = Z∗•(Sscroll|Mscroll)This means that, once the user enters the website,he/she can do as many semantic zooms aswanted and then moves to the next part of thevisualization by clicking the scroll label or bymouse.

The main visualization (Figure 3) expands theY-Axis of the previous one allowing a more clearview of the different lines. It adds eight filters inthe form of circles on the top right corner of thevisualization, a description on how to read theinformation which appears on the bottom rightcorner and a description of the information onscreen, in the top left corner. The Click Scroll

interaction is no longer available but Mouse Scrollis possible. Hover over line and Go through thedetail are available. The new low-level interactionsare:

• How to Read: When the user places themouse over the text in the bottom right cornerof the visualization a brief explanation of howthe visualization works appears. We will usethe symbol R for this. It is important tonotice that this interaction is only availablewhen the first filter is selected. The remainingfilters remove the legend.

• Hover Labels: Each filter is accompanied bya description of the data on screen. On eachdescription, except for the first filter andfourth filters, there are labels that open thesmall visualization described earlier. Thisaction happens when the user places themouse over the labels. This interaction willbe represented by Hlabels.

• Click Next: Each description that appearson each filter contains a “Next” label. Whenthe user clicks on it, the visualization movesto the next filter. We will use the symbolCnext for this. We will add a new symbolN = (Cnext|Mscroll).

• Filters: The eight available filters allowthe user to remove some elements from thevisualization in order to highlight others.There are seven group of industries, each filtershows one of those groups. The eighth oneshows all the industries (Figure 4). Whenthe first filter is selected, the user can do theZ and R interactions; when the fourth filteris selected the user can only do Z. Whenthe other filters are selected the user canZ and Hlabels. The user can change thevisualization by selecting a different filter orby Cnext or Mscroll. When the user doesCnext visualizing the last filter on screen, thevisualization changes from the main one tothe detailed one. The symbols for the filterswill be F1, F2, F3, F4, F5, F6, F7 and F8.In order to simplify the SCI we will use thesymbol f = F2|F3|F5|F6|F7.

The SCI for the main visualization isVmain = (F1 •(Z |R)∗ •N |F4 •Z∗ •N |f •(Z|Hlabels)∗ •N)∗ •F8 •(Z |Hlabels)∗ •N

This means that if the user chooses the first filterthen Z and R are the available interactions, if thefourth filter is selected then only Z is possible.Filters 2, 3, 5, 6 and 7 allow the user to Z andHlabels. The eighth filter is handled differentlybecause when the user uses the N interaction, hemoves to the detailed visualization.


45

The final part of this visualization is the detailedone (Figure 5). This visualization contains all thesmall visualizations accessible in the main part.There are two interactions available at this point,Go through the detail and Mouse Scroll. There areno new interactions possible at this point. TheSCI for the last part is Vdetailed = (D|Mscroll)∗.This means that the user can scroll detailedvisualizations using the mouse or go through thedetails on each small visualization.

The entire visualization is described by the SCIV = Vinitial•Vmain•Vdetailed. This isV = Zoom∗ •(Sscroll |Mscroll) •(F1 •(Z |R)∗

•N |F4 •Z∗ •N |f •(Z |Hlabels)∗ •N)∗ •F8 •(Z|Hlabels)∗ •N •(D |Mscroll)∗

Figure 4: Eight filters are available on the mainvisualization. Each one allows the user to visualizea subset of industries.

Figure 5: All the small visualizations are listed inthe final part of the website. These visualizationsare divided into industrial categories.

4.2.1 Coverage Criteria for Valid Se-quences

The minimum sequence of interaction valid for thisvisualization is 3. A scroll as the first interaction,then the selection of the last filter on screen andfinally, a click on the label “Next” or a mousescroll. The following test set T3 satisfies the BaseCoverage criteria. No errors were found whenexecuting T3.

T3 = {Cscroll •F8 •Cnext, Mscroll •F8 •Cnext,Cscroll •F8 •Mnext, Sscroll •F8 •Mnext }

Valid sequences of length 4 are possible and thefollowing test set T4 contains all of them, satisfyingthe Base+1 Coverage criteria. No error were found

when executing T4. There are valid sequences oflength 5 but they are too many to list in thisarticle.

T4 = {Cscroll •F8 •Cnext •D, Mscroll •F8•Cnext •D, Cscroll •F8 •Mnext •D, Sscroll •F8•Mnext •D, Cscroll •F8 •Cnext •Mscroll, Mscroll

•F8 •Cnext •Mscroll, Cscroll •F8 •Mnext Mscroll,Sscroll •F8 •Mnext •Mscroll, Hline •Cscroll •F8•Cnext, Hline •Mscroll •F8 •Cnext, Hline •Cscroll

•F8 •Mscroll, Hline •Cscroll •F8 •Mscroll}

4.2.2 Coverage Criteria for Invalid Se-quences

The following test set Ti1 contains all the invalidsequences of length 1 for the current visualization.None of these sequences are possible when tryingto execute them in the website. Notice that Cscroll

and Mscroll are not present in the test set becausethey are valid first interactions.

Ti1 = {Hline, D, R, Cnext, Hlabels, F1, F2, F3,F4, F5, F6, F7, F8 }

A test set containing the invalid sequences oflength 2 will have close to 200 elements. As inthe case of the valid sequences of length 5, it is toextensive to include in this article.

4.2.3 Test Cases Remarks

The last test cases showed the need for a tool toassist in the creation of the test sets. For someSCI the amount of sequences of interaction thatcan be derive from it can grow exponentially. It isour goal to develop a tool to create test sets basedon the SCI and Coverage Criteria.

5 Conclusions and future work

In this paper we have described a black-boxtechnique for testing information visualizationimplementations. The technique is based on theconstraints found in the order in which low-levelinteractions must be used in the visualization.We presented several Coverage Criteria for validand invalid sequences. Black-box testing onvisualizations is an unexplored area, one that mustbe considered in order to ensure the quality ofany visualization implementation. Our goal isto start with the development of test methodsfor visualizations, adapted for the needs andrequirements of the area. These methods thatcan be used to provide a rigorous evaluation ofeach visualization.

Future research must include the developmentof a framework to test visualizations using theproposed technique. Such framework should beable to interact with any implementation forruntime verification, that is to check the orderin which interaction are used at runtime against


46

the generated grammar. The framework mustalso generate test cases based on the SCI forthe tester to used, valid and invalid sequencesusing the Coverage Criteria. We must continueinvestigating in order to achieve a framework withthe full support of automated testing.

Finally, the proposed technique should beextended to include parameterized interactions.As it was presented in this article, a SCI candescribe that, for example, before a Detail ondemand the user must do a Selection, but nowhereit is said that the Detail on demand is performedon the items selected by the Selection interaction.This relation may seem trivial now, but aparametrized technique will be more expressive,and as such more powerful.

Acknowledgment

This work was partially supported by thefollowing research projects: PGI 24/N037 andPGI 24/ZN29 from the Secretarıa General deCiencia y Tecnologıa, Universidad Nacional delSur, Argentina.

References

[1] Kirby, Robert M., and Claudio T. Silva. “Theneed for verifiable visualization”. ComputerGraphics and Applications, IEEE 28.5, 78-83,2008.

[2] Banerjee, Ishan, et al. ”Graphical userinterface (GUI) testing: Systematic mappingand repository.” Information and SoftwareTechnology 55.10, 1679-1694, 2013.

[3] Chang, Tsung-Hsiang, Tom Yeh, and RobertC. Miller. ”GUI testing using computer vision.”Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems. ACM,2010.

[4] Hartson, H. Rex, Antonio C. Siochi, andDeborah Hix. ”The UAN: A user-oriented rep-resentation for direct manipulation interfacedesigns.” ACM Transactions on InformationSystems (TOIS) 8.3, 181-203, 1990.

[5] Appert, Caroline, Michel Beaudouin-Lafon,and Wendy E. Mackay. ”Context matters:Evaluating interaction techniques with the CISmodel.” People and Computers XVIIIâĂŤDe-sign for Life. Springer London, 279-295, 2005.

[6] A. C. Siochi and H. R. Hartson. Task-orientedrepresentation of asynchronous user interfaces.In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems (CHI’89), K. Bice and C. Lewis (Eds.). ACM, NewYork, NY, USA, 183-188, 1989.

[7] Ferriday, C. “A Review Paper on DecisionTable-Based Testing”. Swansea University,CS339-2007, 2007.

[8] Seo, K. I., Choi, E. M. “Comparison of fiveblack-box testing methods for object-orientedsoftware”. In Software Engineering Research,Management and Applications, 213-220, 2006.

[9] Khan, Mohd Ehmer. “Different approaches towhite box testing technique for finding errors”.International Journal of Software Engineeringand Its Applications 5.3, 1-14, 2011.

[10] Kovalerchuk, Boris, and James Schwing.“Visual and Spatial Analysis”. Advances InData Mining, Reasoning, And ProblemSolving, 2005.

[11] Keller, Peter R., and Mary M. Keller. ”Visualcues: practical data visualization”. Vol. 2. LosAlamitos, CA: IEEE Computer Society Press,1993.

[12] Shneiderman, Ben. ”The eyes have it: Atask by data type taxonomy for informationvisualizations.” Visual Languages, 1996. Pro-ceedings., IEEE Symposium on. IEEE, 1996.

[13] Jorgensen, Paul C. Software testing: acraftsman’s approach. CRC Press, 2013.

[14] Weileder, Stephan. Test models and coveragecriteria for automatic model-based testgeneration with UML state machines. Diss.Humboldt University of Berlin, 2010.

[15] Friske, Mario, Bernd-Holger Schlingloff, andStephan WeiÃ§leder. ”Composition of Model-based Test Coverage Criteria.” MBEES. 2008.

[16] Yuan, Xiaojun, Xiangmin Zhang, and AlexTrofimovsky. ”Testing visualization on the useof information systems.” Proceedings of thethird symposium on Information interactionin context. ACM, 2010.

[17] Koshman, Sherry. ”Testing user interactionwith a prototype visualizationâĂŘbased in-formation retrieval system.” Journal of theAmerican Society for Information Science andTechnology 56.8, 824-833, 2005.

[18] Aerts Jeroen , Keith C. Clarke, and AlexD. Keuper. ”Testing popular visualizationtechniques for representing model uncertainty.”Cartography and Geographic InformationScience 30.3, 249-261, 2003.

[19] Etiene, Tiago, et al. ”Verifiable visualizationfor isosurface extraction.” IEEE Transactionson Visualization and Computer Graphics 15.6,1227-1234, 2009.


47

[20] Etiene, Tiago, et al. ”Topology verificationfor isosurface extraction.” IEEE Transactionson Visualization and Computer Graphics 18.6,952-965, 2012.

[21] Etiene, Tiago, et al. ”Verifying volumerendering using discretization error analysis.”IEEE transactions on visualization andcomputer graphics 20.1, 140-154, 2014.

[22] Hellmann, Theodore D., Ali Hosseini-Khayat,and Frank Maurer. ”Agile interaction designand test-driven development of user inter-facesâĂŞa literature review.” Agile SoftwareDevelopment. Springer Berlin Heidelberg, 185-201, 2010.

[23] Memon, Atif M., and Bao N. Nguyen.”Advances in automated model-based systemtesting of software applications with a GUIfront-end.” Advances in Computers, 121-162,2010.

[24] Kirani, Shekhar H., and W. T. Tsai. Spec-ification and verification of object-orientedprograms. Diss. University of Minnesota, 1994.

[25] Yi, Ji Soo, et al. ”Toward a deeperunderstanding of the role of interaction ininformation visualization.” Visualization andComputer Graphics, IEEE Transactions on13.6, 1224-1231, 2007.

[26] Daniels, F. J., and K-C. Tai. ”Measuring theeffectiveness of method test sequences derivedfrom sequencing constraints.” Technology ofObject-Oriented Languages and Systems, 1999.TOOLS 30 Proceedings. IEEE, 1999.

[27] Havre, Susan, Beth Hetzler, and Lucy Nowell.”ThemeRiver: Visualizing theme changes overtime.” Information Visualization, 2000. InfoVis

2000. IEEE Symposium on. IEEE, 2000.


48

black-box testing technique for information visualization

Documents