XCheck, a benchmark checker forXML query processors

Enrico Zimuel

zimuel@science.uva.nl, ezimuel@sci.unich.it

Informatics Institute, University of Amsterdam

Dipartimento di Scienze, Universita’ ”G.D’Annunzio” Chieti - Pescara (Italy)

XCheck, a benchmark checker for XML query processors – p.1/32

Contents

Benchmarking of XML engines

What’s XCheck?

How to obtain reliable program runtimes

Query elaboration times

The architecture of XCheck

The XML engines already supported

Running phase

Data analysis phase

Examples of benchmarks

References

XCheck, a benchmark checker for XML query processors – p.2/32

Benchmarking

In computing, a benchmark is a method to assess therelative performance of an object, by running a numberof standard tests and trials against it.

The practice of benchmarking can help vendors,developers, and users to evaluate the performance ofan object.

Moreover, it can help researchers to spot theweaknesses of the current technology and hence topropose improved solutions.

Benchmarking is particularly important when atechnology is young. This is the case of XML.

XCheck, a benchmark checker for XML query processors – p.3/32

Benchmarking (2)

Running a benchmark on different applications takes alot of time and usually generates huge amounts of rawdata.

Interpreting the outcomes of the evaluation is a subtleand crucially important task.

Moreover a single execution of an experiment is notsufficient to obtain reliable results. It’s often necessaryto run the experiment n times in order to reduce thestandard deviation of the results.

XCheck, a benchmark checker for XML query processors – p.4/32

XCheck

XCheck is an open source software for automaticexecution of a benchmark on XML query processors(XPath and XQuery processors).

XCheck works in two phases: running and dataanalysis.

In the running phase XCheck executes the benchmarkon available XML engines and stores the engineexecution times that it measures itself, as well as theexecution times that the engines output.

Optionally, it also stores the benchmark query results.

XCheck, a benchmark checker for XML query processors – p.5/32

XCheck (2)

In the data analysis phase XCheck elaborates somestatistics on the execution times gathered in the runningphase.

As a result it generates performance reports (in XMLand HTML formats) containing lots of plots (inPostScript and PNG formats).

XCheck focuses on performance benchmarks, asopposed to correctness benchmarks. Thus, thebenchmarks do not need to specify correct answers.Nevertheless, XCheck helps to detect incorrect answersby comparing the sizes of the query results obtainedfrom different engines.

XCheck, a benchmark checker for XML query processors – p.6/32

Technical specification

XCheck works with a Gnu/Linux operating system. Intheory is possibile to use XCheck with anotheroperating systems but we have tested it only ondifferent versions of Gnu/Linux.

XCheck is written in Perl and you need a Perlinterpreter, ver. 5.8.5 or higher, installed on yourcomputer in order to execute it. There are also twoCPAN Perl modules you need to install.

If you want to generate the plots of the benchmark youalso need the Gnuplot software ver. 4.0 or higher.

XCheck is released under the GNU General PublicLicense

XCheck, a benchmark checker for XML query processors – p.7/32

Reliable program runtimes

Measuring reliable program runtimes is not asstraightforward as it may appear. Computer’s hardwareand system software influence the runtime of aprogram.

The CPU speed, the amount of main memory, theamount of cache, the operating system and thecompiler all play significant roles.

For Java applications, the Java virtual machine imposesanother level of software between the program and theoperating system that may alterate the runtimes of theapplications under evaluation.

XCheck, a benchmark checker for XML query processors – p.8/32

Reliable program runtimes (2)

To avoid unreliable results, XCheck runs the sameexperiment n + 1 times and takes the average and thestandard deviation of the last n evaluation times.

The bigger is the value of n, the less is the impact onthe evaluation time of the warm-up time and of othercosts that are external to the query evaluation.

The value of n can be specified with a command lineoptions of XCheck.

XCheck, a benchmark checker for XML query processors – p.9/32

CPU time vs. wall clock time

Besides collecting the times that the engines report,XCheck itself measures the query total execution time

This time is reported in CPU time. The CPU time (orprocessor time) is the amount of time the processoractually spends running a program.

The other times are engine dependent and can be CPUtime as well as wall clock time. The wall clock time isthe elapsed time between when a process starts to runand when it is finished.

It is important to remember this fact when comparingthe performance of different engines, since, in principle,CPU time and wall clock time should not be compared.

XCheck, a benchmark checker for XML query processors – p.10/32

Query elaboration times

XCheck uses the following types of query elaboration times:

document processing time is the elaboration time of theXML document.

query compile time is the elaboration time for thetranslation of the query in the internal language or datastructure of the engine.

query execution time is the engine elaboration time ofthe query.

serialization/output time is the engine elaboration timefor generate the file containing the output of the queryresult.

total time is the response time of the engine. This timeis always CPU time.

XCheck, a benchmark checker for XML query processors – p.11/32

The architecture of XCheck

XCheck, a benchmark checker for XML query processors – p.12/32

The XML engines already supported

Engines Version Type TimesSaxonB 8.7 XQuery D,QC,QE,TGalax 0.5.0 XQuery TMonetDB/XQuery 0.10.3 XQuery D,QC,QE,S,TQizx/open 1.0 XQuery QE,S,TeXist 1.0 XQuery TQexo 1.8.1 alpha XQuery TBlixem 16 Jun 2005 LiXQuery TXmlTaskForce 30 Sep 2004 XPath 1.0 TArb ? CoreXPath D,QE,T

where D=document processing time, QC=query compile time, QE=query execution time,S=serialization/output time and T=total time.

XCheck, a benchmark checker for XML query processors – p.13/32

Running phase

An experiment consists of running a set of XML engineson a set of queries and a set of input XML documents.

The user can specify the engines, the queries and thedocuments that participate in the experiment in an xmlfile, named experiment.xml.

You can execute an experiment in the running phase ofXCheck with the following command:

$ ./XCheck.pl -run exp1

where exp1 is the name of the directory, under thedirectory experiments, where XCheck looks for theexperiment specification file experiment.xml.

XCheck, a benchmark checker for XML query processors – p.14/32

Running phase (2)

During the running phase XCheck prints on thestandard output some informations about the executionof the experiment. For each query XCheck uses thestring (ok) to indicate an execution witout error and thestring (!?) to indicate error.

At the end of execution XCheck produces two files,named outcome.xml and outcome.html.

These files are stored in the directory/experiments/exp1/output/ of XCheck.

XCheck, a benchmark checker for XML query processors – p.15/32

Running phase (3)

As default option XCheck executes each query 4 timesand takes the average and the standard deviation of thelast 3 runs.

If all four executions of a query produce errors XCheckstores the error and goes to the next query. If at leastone execution gives results XCheck stores them.

You can change the number of executions with the useof the option -n num. For instance, you can executethe queries one time (+1) with the following command:

$ ./XCheck.pl -run exp1 -n 1

XCheck, a benchmark checker for XML query processors – p.16/32

Running phase (4)

The use of the -p option tells XCheck to generatediverse plots presenting the experiment executiontimes. For instance:

$ ./XCheck.pl -run exp1 -p

After the execution of the experiment exp1 XCheckgenerates several types of plots and places them insubdirectories of /experiments/exp1/output/.

All the plots are also linked in the main report fileoutcome.html, so that after the execution of therunning phase the user is able to see all the filesgenerated by XCheck viewing only outcome.html.

All the HTML files generated by XCheck are in standardW3C HTML 4.01

XCheck, a benchmark checker for XML query processors – p.17/32

Example of plots

10

100

1000

QizxGalaxSaxonBMonetDB

Log

Ave

rage

Exe

cutio

n T

ime

(sec

)

Engine

XCheck output (running phase), Benchmark: XMark, Average Execution Time

0 2 4 6 8 10 12 14 16 18

Plot 3D, MonetDB, Total execution time (sec), Benchmark: xmark

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21Query 0

1 2

3 4

5 6

7 8

Document

0

2

4

6

8

10

12

14

16

Time (sec)

0.01

0.1

1

10

100

1000

q20q19q18q17q16q15q14q13q12q11q10q9q8q7q6q5q4q3q2q1

Log

Que

ry e

xecu

tion

time

(sec

)

Query

XCheck output (running phase), Benchmark: XMark, Document: d1.024 (113.99 MB)

MonetDBSaxonB

Qizx

0.01

0.1

1

10

100

1000

q20q19q18q17q16q15q14q13q12q11q10q9q8q7q6q5q4q3q2q1

Log

Que

ry e

xecu

tion

time

(sec

)

Query

XCheck output (running phase), Benchmark: XMark, Document: d1.024 (113.99 MB)

MonetDBSaxonB

Qizx

XCheck, a benchmark checker for XML query processors – p.18/32

Running phase (5)

By default, XCheck doesn’t store the query results forthe reason of saving disk space. The user can see onlythe size of the results in bytes in outcome.html

If you want to store the results of the queries you needto use the option -s of XCheck, for instance:

$ ./XCheck.pl -run exp1 -s

All the query results are stored in the directoryexperiments/exp1/output/results. The result files are also linked inoutcome.html, in the last table of the report page.

XCheck, a benchmark checker for XML query processors – p.19/32

The update option

After the execution of an experiment if you want toinsert new xml documents, queries or engines into theexperiment without repeating the whole experimentfrom the beginning, you can use the update option.

With this option XCheck keeps the old results of theexperiment and executes only the new input. In this waythe you don’t lose time with the re-execution of the oldinput.

In order to execute the update option you must use thefollowing command:

$ ./XCheck.pl -run exp1 -update

where exp1 is the name of the experiment.

XCheck, a benchmark checker for XML query processors – p.20/32

Data analysis phase

After the execution of an experiment XCheck can helpthe user to analyze the results with the use of someaggregation measures, statistics and plots.

The input of the data analysis phase is the output of therunning phase, the file outcome.xml to be precise.

XCheck uses the following aggregation measures: sum,mean, median, minimum, maximum, standarddeviation, medley relay data speed , medley relay queryspeed , data scalability factor , query scalability factor ,similarity index .

XCheck, a benchmark checker for XML query processors – p.21/32

Data analysis phase (2)

We define the medley relay data speed of Ek on thedocument set A and the query set B, denoted bymrdskA,B, as follows:

mrdskA,B =|B| ·

∑i∈A sizei

∑i∈A,j∈B tki,j

We define the medley relay query speed of Ek on thedocument set A and the query set B, denoted bymrqskA,B, as follows:

mrqskA,B =|A| ·

∑i∈B lengthi∑

i∈A,j∈B tki,j

XCheck, a benchmark checker for XML query processors – p.22/32

Data scalability factorWhenever data scalability is a benchmark target, thefollowing definition of data scalability factor becomesrelevant. Let I = (i1, i2, . . . , ik), for k ≥ 2, be a sequenceof indexes of documents of increasing sizes. We definethe data scalability factor of Ek on the documentsequence I and the query set B, denoted by dskI,B, asfollows. If k = 2, then

dsk(i1,i2),B =mrdski1,B

mrdski2,B

If k > 2, then

dskI,B =

∑k−1j=1 dsk(ij ,ij+1),B

k − 1XCheck, a benchmark checker for XML query processors – p.23/32

Query scalability factorSimilarly, when query scalability is a benchmark target,the following definition of query scalability factorbecomes relevant. Let J = (j1, j2, . . . , jk), for k ≥ 2, be asequence of indexes of queries of increasing lengths.We define the query scalability factor of Ek on thedocument set A and the query sequence J , denoted byqskA,J , as follows. If k = 2, then

qskA,(j1,j2)=

mrqskA,j1

mrqskA,j2

If k > 2, then

qskA,J =

∑k−1i=1 dskA,(ji,ji+1)

k − 1XCheck, a benchmark checker for XML query processors – p.24/32

Scalability factorNotice that, by virtue of the definition of medley relayspeed, a scalability factor (either for data or query) lessthan 1 (respectively, equal to 1, bigger than 1)corresponds to a sub-linear (respectively, linear ,super-linear ) increase of the total evaluation time whenmoving from one instance of the problem to a biggerone.

0

0.5

1

1.5

2

2.5

q20q19q17q16q15q14q13q10q9q8q7q6q5q4q2q1

dsfa

ctor

Query

Index S3, dsfactor (total_time), Experiment: xmark

MonetDBSaxonB

GalaxQizx

Linear scalability

XCheck, a benchmark checker for XML query processors – p.25/32

Similarity index

XCheck uses a similarity index to compare the similarityof the behaviour of two engines.

Let X = (x1, x2, . . . xn) be a sequence containing theresults on n experiments. For each 1 ≤ i ≤ n − 1, letW i

X = a1a2 . . . an−i where, for each 1 ≤ j ≤ n − i, wehave that aj = U if xi < xi+j, aj = D if xi > xi+j, andaj = E if xi = xi+j.

Let WX = W 1X ◦ W 2

X . . . ◦ Wn−1X be the similarity word

associated to X, where ◦ is the concatenationoperation.

XCheck, a benchmark checker for XML query processors – p.26/32

Similarity index (2)

Given two sequences X = (x1, x2, . . . xn) andY = (y1, y2, . . . yn), the distance between X and Y isdefined as the word distance between the words WX

and WY , where the word distance between two wordsof the same length is the number positions of the twowords that contain a different character.

For instance, the word distance between UUED andEUEU is 2 (the characters in both the first and lastposition differ in the two words).

The similarity index is defined as the distance betweenX and Y over n, the length of the sequences. Lowvalues of the index means best similarity (0= best, 1=worse).

XCheck, a benchmark checker for XML query processors – p.27/32

Statistics indicesXCheck generates six types of statistics indices:

S1 The elaboration of the aggregation measure indicescomputed on each engine, query and document.

S2 The elaboration of the aggregation measure indices foreach document.

S3 The elaboration of the aggregation measure indices foreach query.

S4 The plots in 3d for each engine with differentelaboration times.

S5 The elaboration of the similarity index for eachdocument and pair of engines.

S6 The elaboration of the similarity index for each queryand pair of engines.

XCheck, a benchmark checker for XML query processors – p.28/32

Statistics indices (2)

For each statistics indices XCheck generates two filesin XML and HTML formats.

In the data analysis phase the user can also generatedifferent plots of the elaboration times with differentaggregations of the data.

The user can choose to generate these plots: plots oftimes for each document (P1), plots of times for eachquery (P2), plots of times for each engine anddocument (P3), plots of times for each engine andquery (P4).

XCheck, a benchmark checker for XML query processors – p.29/32

Data analysis phase (3)

In order to execute the data analysis phase the usermust specify the engines, the documents, the queries,the type of elaboration times and the aggregationmeasures to be used in the elaboration.

Moreover the user can specify the type of statistics(S1,...,S6) and plots (P1,...,P4) that should begenerated.

All these informations are reported in theanalysis.xml file stored in the directory of theexperiment.

In order to execute data analysis phase you can use thefollowing syntax:

$ ./XCheck.pl -data exp1

XCheck, a benchmark checker for XML query processors – p.30/32

Examples of benchmarks

We executed the following benchmarks with the use ofXCheck:

XMark, http://monetdb.cwi.nl/xml/

Michigan, http://www.eecs.umich.edu/db/mbench/description.html

XMach-1,http://dbs.uni-leipzig.de/en/projekte/XML/XmlBenchmarking.html

X007, http://www.comp.nus.edu.sg/~ebh/XOO7.html

XBench,http://se.uwaterloo.ca/~ddbms/projects/xbench/index.html

All the results of these benchmarks are available on theXCheck web site.

XCheck, a benchmark checker for XML query processors – p.31/32

ReferencesXCheck official web site, http://www.xcheck.org

M. H. Kay, Saxon. An XSLT and XQuery processor,http://saxon.sourceforge.net

M. Fernández et al., Galax. The XQuery implementation for discriminating hackers,http://www.galaxquery.org

CWI Amsterdam, MonetDB/XQuery. An XQuery Implementation,http://monetdb.cwi.nl/XQuery

Axyana software, Qizx/open. An open-source Java implementation of XQuery,http://www.axyana.com/qizxopen

Wolfgang Meier, eXist. Open Source Native XML Database,http://exist.sourceforge.net

Qexo - The GNU Kawa implementation of XQuery,http://www.gnu.org/software/qexo/

University of Antwerp, Blixem. LiXQuery engine.,http://adrem.ua.ac.be/~blixem/

XML TaskForce XPath, http://www.xmltaskforce.com

Christoph Koch, Arb, http://www.infosys.uni-sb.de/~koch/projects/arb/

XCheck, a benchmark checker for XML query processors – p.32/32