the hrmes universal manual

The Hrmes FrameworkA data-driven test framework for Selenium RC

The Universal Manual for Hrmes

of 66

Document Title

Hrmes User Manual

Document Abstract

Hrmes is the Human Readable Machine Executable Specification(pronounced Hermes) Test Framework designed and developed by Andrew Thompson and W3QA Ltd.This document describes the procedures for installing and configuring Hrmes. Additionally it outlines how to run, develop and understand its reporting features. This document will be in-line with the code release of the Hrmes project.

Document History

Date Hrmes Version

Purpose of Revision Author

14/02/2009 Hrmes_0_1_0 This revision supports the initial release Hrmes. Andrew Thompson


Contents

1 Introduction............................................................................................5

2 The Hrmes License ................................................................................6

3 The Hrmes Framework Pre-requisites ........................................................7

3.1 Step 2 – Install Eclipse...............................................................................73.2 Step 3 – Install Firefox 2 with appropriate plug-in..............................................8

4 Installation and Configuration of the Hrmes Framework.....................................9

4.1 Installing through Eclipse...........................................................................94.2 Installing Additional Eclipse Plug-ins - Optional...............................................124.3 Remaining Configuration..........................................................................13

5 The Hrmes Format and the Selenium IDE......................................................14

5.1 The Hrmes Selenium IDE Format.................................................................145.2 The Hrmes Format File Conversion..............................................................215.3 The Hrmes Format Files...........................................................................22

6 Running Hrmes.......................................................................................25

6.1 Test Execution via Eclipse.........................................................................256.2 Test Suites...........................................................................................266.3 The Hrmes Test Methodology.....................................................................27

7 Demo - An End to End Example .................................................................29

7.1 Record ...............................................................................................297.2 Export and Convert.................................................................................307.3 Run and Report......................................................................................31

8 Using CVS through Eclipse.........................................................................33

9 Understanding the Hrmes Framework Reports................................................36

10 Developing the Hrmes Framework.............................................................39

10.1 The Application mappings.......................................................................3910.2 Java Custom Methods............................................................................42

11 Selenium GRID and the Hrmes Framework...................................................45

11.1 Selenium GRIDs integration with the Hrmes Framework...................................4511.2 Selenium GRID and its Hrmes Framework Ant scripts......................................4711.3 Selenium GRID integration with the Hrmes Framework....................................48

12 Release Management and the Hrmes..........................................................50

12.1 Upgrading To Newer Versions...................................................................5012.2 The Impact On Shared Data From Upgrades..................................................51

13 The Hrmes Frameworks Database Schema...................................................53

13.1 Building the Hrmes Schema......................................................................5313.2 Loading Test Case Data into the Hrmes Schema............................................54

of 66


13.3 Loading Test Results into the Hrmes Schema................................................5413.4 Using the SQL Explorer Eclipse Plug-in........................................................5413.5 Database Run Mode...............................................................................5513.6 Graphical Analysis................................................................................55

14 Debugging tests with the Hrmes Framework................................................57

14.1 Debugging...........................................................................................5714.2 Debugging through Eclipse.......................................................................5714.3 Debugging through the IDE.......................................................................5914.4 Debugging using Hrmes...........................................................................5914.5 Debugging Manually...............................................................................60

15 Troubleshooting the Hrmes Framework.......................................................61

15.1 Ants CLASSPATH settings.........................................................................6115.2 Hrmes Frameworks Java CLASSPATH settings................................................6115.3 Hrmes Frameworks Source......................................................................6215.4 The S-IDE Export Process........................................................................6315.5 The Hrmes Framework Source within the Selenium-IDE ..................................6315.6 S-IDE Export and Extraction Failure ...........................................................64

16 Support for the Hrmes Framework.............................................................66

16.1 Use SourceForge to Raise Support Requests.................................................66

of 66

1 IntroductionThis is the user manual for the Human Readable Machine Executable Specification (Hrmes – pronounced Hermes) Selenium RC Test Framework. It will take you through its installation, configuration, development and execution.

The Hrmes framework is a test automation framework built on the OSS Selenium RC (SRC) platform. It is designed for use on all web based applications.

Hrmes enables a unified process for test specification, execution and reporting. It has a clear domain specific language for test specification which supports manual and automated testing through the same singular Hrmes Format asset.

The structure and narrative of Hrmes is based closely on Seleniums’ Fitnesse. All test assets produced via the Selenium IDE can be easily converted into Hrmes tests assets.

Hrmes is the first Selenium framework to achieve a data driven approach on a fully enabled Selenium RC platform.

Hrmes is fully integrated with Selenium GRID. Out of the box you can run distributed tests on the platform configurations of your choice - in the same way you could using Selenium GRID directly.

Hrmes provides a powerful and usable set of tools. It facilitates a standard test process and a singular environment for all test assets across all projects.

The Hrmes Framework gives the general test community a scalable framework for developing and rolling out structures, processes and software for itself and its clients.

Automation frameworks cannot mitigate the need for technical resources on a test automation project. All such projects are technical by their nature. The Hrmes project goes a long way to enable the full and continuous involvement of non technicians in a Selenium RC automation project. However, Hrmes does require some technical ability and knowledge of Selenium to fully exploit. Hrmes will work best with teams that have access to test engineering resources with some programming experience.

The Hrmes Framework provides increased clarity, usability and accessibility to a Selenium RC automation project, without diluting any of its power.

of 66

2 The Hrmes License The Hrmes Framework is available on an Apache 2 license. It is assumed that you have read and agreed to these licensing terms.

FURTHERMORE: THE Hrmes FRAMEWORK SOFTWARE IS PROVIDED "AS-IS" WITHOUT ANY WARRANTY WHATSOEVER. YOU ASSUME ALL RISKS AND RESPONSIBILITIES FOR SELECTION OF THE SOFTWARE TO ACHIEVE YOUR INTENDED RESULTS, AND FOR THE INSTALLATION OF, USE OF AND RESULTS OBTAINED FROM THE SOFTWARE. THE POINT CLARK NETWORKS MAKES NO WARRANTY THAT THE SOFTWARE WILL BE ERROR FREE OR FREE FROM INTERRUPTION OR FAILURE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, W3QA LTD DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT WITH RESPECT TO THE SOFTWARE AND

of 66

3 The Hrmes Framework Pre-requisites The Hrmes Framework can be installed on any operating system that supports Java,The Hrmes test data resides in a spreadsheet format. This manual assumes that you are using MS Excel. However, Open Office or any other spreadsheet package will work equally well.

You need to install the following before running the Hrmes Framework:

Hrmes Framework Pre-requisites

Software Version Location Size

Eclipse Eclipse IDE for Java developers - 3.2.1 (aka - Europa)

http://www.eclipse.org/downloads/ 80MB

Mozilla Firefox Browser

2.0 – Not 3.0 http://www.mozilla.com/en-US/Install Selenium IDE plugin (v1.0)for from Firefox:https://addons.mozilla.org/en-US/firefox/addon/2079and Firebug:https://addons.mozilla.org/en-US/firefox/addon/1843

If versions listed here are not available choose the closest match.

The Hrmes video need QuickTimePlayer. The video are stored under “Hrmes Framework\Framework\QuickTimeMovies”.

Most of Hrmes’ dependencies are bundled. You will automatically get them when you install Hrmes through Eclipse. This includes the Selenium client and server jars, Java SDK, the Java API for Excel, Java JDBC drivers and other miscellaneous dependencies that you can reference through Eclipse.

The Hrmes project requires about 120MB of disk space. The core Hrmes code itself is less than 100K, but the project does come heavily packaged with almost everything you will ever need for a Selenium based test framework.

Time to download will obviously depend on your bandwidth, but allow about 30 minutes to 1 hour to complete the install and configuration.

3.1 Step 2 – Install Eclipse

1. Using Windows Explorer, create a folder off your C drive and call it Hrmes. 2. Download Eclipse and extract it into Hrmes folder.3. Launch Eclipse (from C:\Hrmes\Eclipse\ folder) and select the C:\Hrmes as

your workspace.4. Create a shortcut and drop this short-cut into your Windows Start menu.

of 66

https://addons.mozilla.org/en-US/firefox/addon/1843




http://www.eclipse.org/downloads/

3.2 Step 3 – Install Firefox 2 with appropriate plug-in

You probably have these, but just in case:

1. Go to the Mozilla site and download Firefox version 2.2. Search for Selenium IDE, Web Developer and Firebug under plug-ins and

install.

of 66

4 Installation and Configuration of the Hrmes Framework

4.1 Installing through Eclipse

The Hrmes Framework code, the vast majority of its dependencies and all related project assets are entirely under source control in the SourceForge CVS repository.

Getting hold of the entire Hrmes project is simple using Eclipse. Within Eclipse, go to Window->Open Perspective -> Other, then select CVS repository.

In the CVS repository tab right-click and select New -> Repository Location.

of 66

Add a new repository with the settings described above.

There is no need to enter a password for the anonymous user, so leave it blank. The anonymous user gives you read access to the repository.

Clicking finish will show you the Hrmes Framework as it is in CVS. Expand the repository so that you can see the “HEAD”, “Branches” and “Versions” entries. It is recommended that you take the latest release. Tagged versions are stable releases, whereas “HEAD” reflects the latest check-in.

Whether you choose “HEAD” or a tagged release from “Versions”, the steps are the same.

of 66

Those settings again:Host: hrmes.cvs.sourceforge.netRepository path: /cvsroot/hrmesUser: anoymous

Important note on Eclipses’ default settingsBefore checking out the project ensure that Eclipse will check out the entire project including all currently empty folders. In Eclipse->Window->Preferences->Team->CVS->Files and Folders, uncheck “Prune empty directories”. To check out the entire framework right click on “Hrmes” and “WebProjects” then select “Check out”.

This action checks out Hrmes onto your hard drive. This will take several minutes.

After check out, Eclipse will alert you to any problems it has via the “Problems” tab. There should be no errors or warnings.

If there are any errors – you will have to resolve them before continuing. Have a look at the “Trouble Shooting” section at the end of this document if you run into problems. You can now view the code base of the Hrmes Framework. You do this via the Java perspective (Window-> Open Perspective -> Java).

of 66

4.2 Installing Additional Eclipse Plug-ins - Optional

You can skip this section if you do not wish to use CubicTest right now. It is not required by Hrmes Framework and can be installed anytime.

In Eclipse go to Help->Software Updates -> Find and Install.

Change the default to search for new features. After clicking new remote site you will be presented with the following:

of 66

CubicTest settings for the update manager should be set accordingly (URL: http://www.cubictest.org/update):

One of many useful things about Eclipse is that you can manage plugin upgrades easily through it.

4.3 Remaining Configuration

The Demo project has been created to provide a basis for exploring what the framework can do. It is also template for what a Hrmes project should look like initially.

Build a project for your application using the Demo project as a template. Using copy and paste in Eclipse, is probably the easier way.

The remaining configuration is based on the run-time properties of Hrmes. They are best explained in that context.

of 66

http://www.cubictest.org/update

5 The Hrmes Format and the Selenium IDE

You can build your test scripts directly in the Hrmes format. You may also convert Selenese tests created through the Selenium IDE into this format.

If you build your scripts directly in the Hrmes format, you will not have to wait for your AUT to be available to start writing your test scripts.

The ability to abstract test definitions with Hrmes gives you more flexibility on how you compose your test scripts.

For instance you may write dozens of test cases with a reference to a target called “search box”, before the AUT is delivered. Subsequently, all you need do is create a mapping rather than rewrite all your test cases.

In this way Hrmes, supports an important Agile principle of “test early”.

Hrmes may also be used as a common format for manual and automation tests. This will minimise the reworking of manual test assets when they need to get automated.

When you compose your tests directly in the Hrmes format, use the Selenium IDE as much as the state of your application will allow. The Selenium IDE will be a help when you need to identify your applications various locators and to remind you of the pretty extensive Selenium command set. All of which can be used directly in Hrmes.

You can skip the next 2 sections if you are not intending to convert tests from the Selenium IDE into the Hrmes format.

5.1 The Hrmes Selenium IDE Format

The Selenium IDE supports the export of its native Selenese HTML into several formats. We are now going to add an extra one – the Hrmes Format.

In Eclipse go to Hrmes->Framework->SeleniumIDE->Formats. Open up the file called “Hrmes – Hrmes Format”.

Start Firefox and open the Selenium IDE. Within the IDE open Tools. Select Options -> Options -> Formats.

of 66

Next select “Add” and give the name of the new format “Hrmes - Hrmes Format”.

Keep your Firefox session open and return to your Eclipse session which should have the Hrmes source you viewed earlier. Copy this into the Selenium IDE format source and click “OK”.You should now have something resembling the following:

of 66

The Hrmes format is based on Selenese HTML and facilitates simple conversion from it. The format options allow the user to specify how test data within the Selenium IDE gets parsed into steps by the Hrmes extraction process.

Click “OK” and return to the main Selenium IDE screen. Now you can export any Selenese scripts in the new format. In the Selenium IDE go to File->Export Test Case as-> Hrmes. The “Save as” dialogue will appear and you can now save your export to your project location C:\Hrmes\<Project>\SeleniumIDE\Exported.

You have now exported a HTML file that contains all the information required to convert it into the Hrmes Format. The exported file itself is still almost identical to the original Selenese HTML version. The next step is to convert it into the Hrmes spreadsheet format.

Upload the user-extensions.js file

of 66

This step is optional. Whether you wish to perform it will depend on how match backwards compatibility you wish to preserve between the Hrmes and the Selenium-IDE. This issue is covered later on. You may skip this section if you do not want backwards compatibility.

With the Selenium IDE options window still open click on the “General” tab.

Under the Selenium Core extensions, click the browse button and upload the user-extensions.js file.

Location: “C:\Hrmes\WebProjects\Demo\SeleniumIDE\user-extensions.js”.

of 66

Now click OK. This gives the Selenium-IDE the ability to use custom Javascript methods defined through this file. There is more on what use you can do with this functionality later. Refer to the Selenium website ( www.openqa.org ) for the full explanation on what user extensions are.

If you already make use of user extensions, just merge the contents of the two files. If you do not, I would advise skipping the rest of this section.

Note on S-IDE user extensions

Some user extensions that have been written for the S-IDE will not translate to any Selenium Java API method – or indeed any Selenium RC (SRC) client language method. These are ones that make use of purely S-IDE or Selenium Core objects.

of 66

http://www.openqa.org/

There are few of these. Indeed, I can only recall two - the “goto” and “include” methods.

Whereas many user extensions get incorporated into the core S-IDE language set, these will never make it. They would break the S-IDE code translations, i.e. from Selenese HTML to Java.

The vast majority of pure Javascript user extensions will have some meaning to SRC and can be made available in the Hrmes in one of three ways.

Firstly and probably the simplest way is via custom methods calling getEval. See the development section later on for an explanation. If you have a lot of complex Javascript this can get a bit messy and unreadable.

The other approaches leave the Javascript in the user extension file. They are then available when you start the Selenium server with the –userExtension parameter. This is the default setting for invoking the Selenium server with the Hrmes.

For the user extension method to have any meaning a Java method must be created to handle its use. One way of doing this is by writing a method in the Hrmes class. There is one already created for the user extension “typeRepeated”.

This method acts as a proxy to the Javascript method defined in the user-extension.js file. This is a clean way of making any extension available. However, as the Hrmes class is NOT a web project data file you will have to merge this change every time you do an upgrade of the Hrmes project.

of 66

public void typeRepeated(String locator, String pattern) throws ScreenShotException { /* * This method relies on the doTypeRepeated method being defined as a * user extension. */

try { proc.doCommand("typeRepeated", new String[] { locator, pattern });

} catch (RuntimeException e) { // TODO Auto-generated catch block screenShot(e);

}

}

Lastly, you may directly incorporate your user extensions into the Selenium Java client. In this case the Selenium Java client will need rebuilding.

Although there is some convenience in having a user extension method available directly through the Selenium Java client, it does require the greatest effort of the three approaches.

Firstly you will need to install Mavern and get hold the source code for at least the Selenium client: http://svn.seleniumhq.org/svn/selenium-rc/trunk/ . The Hrmes project has mvn pre-installed and a snapshot of the selenium-rc, circa November 2008. This is provided as no more than an example of what you should end up when you go down this route. See “Hrmes->Mavern”.

You now need to add your new method in iedoc2java.xml and rebuild with Mavern: mvn install.

If you are not comfortable with using Mavern to build the Selenium project, check the Selenium developers’ guide:http://wiki.seleniumhq.org/display/SRC/Developer's+Guide#Developer%27sGuide-DevelopingtheJavaRCClientDriver

If after reading this you find that your comfort level has not been elevated: I would go for either of the first two approaches: getEval or write a method into the Hrmes class.

Incorporating user extensions written for the Selenium IDE or Core, into a Selenium RC platform is a fairly common Selenium challenge.

All three approaches of incorporating pure Javascript user extensions into a Selenium RC framework are discussed on the seleniumhq website in the following thread: http://clearspace.seleniumhq.org/thread/14467 .

Forcing user extensions to live in the world of Selenium RC can be needlessly hard work. Review all the dependencies your test scripts have on user extensions. Look to replace them where you can with custom methods written in Java. Those that cannot be replaced “like for like“, are the ones that will not work on a Selenium RC platform. These will need to be re-worked in other ways – probably by altering the test specifications. This should be rare.

One of the strengths of the Selenium project is that it offers so many entry points for test automation, from the Selenium IDE to Selenium GRID. However, moving from one flavour can be painful. When you move onto the Selenium RC platform from the comfort of the Selenium IDE you do need to “mind the gap”. If your automation project has a lot of dependencies on user extensions this point is especially pertinent.

Of course, if your project does use extensions, then the issue is less relevant and you are probably reading for academic reasons.

of 66

http://clearspace.seleniumhq.org/thread/14467

http://wiki.seleniumhq.org/display/SRC/Developer's+Guide#Developer'sGuide-DevelopingtheJavaRCClientDriver

http://wiki.seleniumhq.org/display/SRC/Developer's+Guide#Developer'sGuide-DevelopingtheJavaRCClientDriver

http://svn.seleniumhq.org/svn/selenium-rc/trunk/

5.2 The Hrmes Format File Conversion

Selenium IDE tests scripts once exported can be converted into the Hrmes format. The conversion process runs via an Ant script called “Start Hrmes Extraction Process”.

If you have not done so already, drop this build file (located under Hrmes/Framework-/AntScripts/Hrmes Extraction) into the Ant Window within Eclipse.

Like all Ant scripts in Eclipse – just double click it when you want it to run. It will process all the Selenium IDE exported files found in your Web Project folder and move them to the neighbouring Extracted folder.

The conversion process will preserve your original file. It will also create a new version of it with an XML file extension and an entirely new Hrmes Format spreadsheet file.

Batch Mode

The conversion process can run against multiple S-IDE Selenese data files at once without the data having been previously exported into the Hrmes format. The batch option is available should you wish to convert a large amount of Selenese data in one run. This is ideal for Selenese test suites.

You need to let the Hrmes know how your test scripts should be parsed. You should use these as you would when you configure the Hrmes export format in the S-IDE. These settings will apply to all test scripts processed in any one batch run. The Hrmes itself will create a S-IDE /Hrmes export version of the raw Selenese file. It then converts it as it would do if you had exported it manually.

From a user’s point of view, the process is almost exactly the same. You need to run the same Ant script and the Selenese data files are expected to be in the S-IDE export location.

You need to let the extraction process know that it’s working in batch mode. So change the side.conversion.mode property to “batch”. By default it is not.

of 66

#################################################################

side.conversion.mode=batchside.parse.token=clickAndWaitside.when.to.parse=after

##################################################################

In batch mode you do not have to configure anything in the S-IDE. You do not even have to add the Hrmes export format to the S-IDE. In batch mode things are driven by the “hrmes.properties” file.

5.3 The Hrmes Format Files

The Hrmes format is entirely derived from the Selenese structure of Command-Target-Value and Selenium‘s version of FIT.

This similarity will minimise any pain adjusting from the Selenium IDE. It also means that testers can avoid the standard route into Selenium RC via a programming language.

The Hrmes format is an enhanced form of Selenese. It can be enhanced further through application mappings (aka GUI mappings) and custom methods.

The Hrmes format facilitates the development of a single test asset, which can be used for manual and automated testing. It is a document that can facilitate auditing and review of test specifications. It provides a clear and easily read document. The flow of the raw Selenium IDE script is transformed into test steps and formatted according to the different type of actions performed.

You may develop your test scripts straight into this format without using the recording features of the Selenium IDE.

Using the Hrmes format directly requires a high level of familiarity with Seleniums command set. If you are new to Selenium, the Selenium IDE is probably a better place to start.

Although spreadsheet packages such as MS Excel have many limitations, they can be easily used to produce sophisticated documentation. They are also a simple and familiar interface to most testers. It is often overlooked how important good documentation is, in keeping managers and to people in the more “business” focused roles in IT involved in an automation project. These are usually the very people who often need to know the detail of what is actually getting tested.

You may use the Hrmes format to abstract all the implementation specific detail you get through the Selenium IDE. This will make your tests easier to read and far easier to maintain. This is dealt with later – under the development section.

Note on Eclipse and Excel

Eclipse will make little effort keeping itself in sync with Excel files on your file-system.

When you open an Excel file it is instantly marked as having been changed (in Eclipse denoted by an asterix to the left of the file name) – despite the fact it has

of 66

not. You can certainly read and write Excel data directly through Eclipse, but do not expect to do much more.

When you will need to do more than this – for instance opening screenshots and stack traces from the Hrmes reports – do so outside of Eclipse, e.g. in Windows explorer.

Preserving backwards compatibility with the Selenium IDE

Using the S-IDE, you may export tests directly into the Hrmes format. You may also import Hrmes test data back into the S-IDE.

During test execution a Selenese data file will be created dynamically. This will correspond directly to the Hrmes test data being run.

The file created is an XML file called seleniumIDE.xml and stored under “WebProjects\SeleniumIDE-\Imported”. This file can be imported into the Selenium IDE and used like any other test script recorded through the S-IDE.

This may require an explanation on why anyone would wish to do this. Test analysts will need to build up a level of comfort using the Hrmes – especially those that are not technical.

The Hrmes project goes further than any other to ease the transition from the world of S-IDE into Selenium RC. However, the learning curve has not been removed completely. Some testers will feel more comfortable knowing their Hrmes tests are fully compatible with the Selenium IDE. It is mainly included as a fall back option.

The task of maintaining compatibility is relatively simple, even when we start extending Hrmes and writing custom methods. Keeping test scripts that have custom methods compatible with the Selenium IDE, means making use of Selenium’s user extension functionality.

of 66

The default Hrmes user-extension.js file that we uploaded earlier, contains the following:

These entries correspond to the basic custom methods of the Hrmes Framework such as “startUp”, “teardown” and “comment”. They are dummy methods that allow any Hrmes test to run within the Selenium-IDE.

To preserve backwards compatibility, this user extension file needs to be kept in line with any custom methods you write. You will need to write a dummy method for each new custom method you write. Hardly onerous, but remember to do so.

As mentioned earlier, preserving backwards compatibility is not a requirement of the Hrmes, but you may find it desirable.

of 66

Selenium.prototype.doStartUp = function() {};Selenium.prototype.doTearDown = function() {};Selenium.prototype.doComment = function() {};

6 Running Hrmes

6.1 Test Execution via Eclipse

Using CVS through Eclipse

Hrmes can be run directly through the Eclipse JUNIT test environment and is setup ready to do so. You may also run your Hrmes Framework scripts through Ant. This is probably the simpler user interface.

The Hrmes framework is built to support system testing of all web projects.

You control the entire Hrmes configuration through property files and each project has its own version appropriate to it under “PropertyFiles”.

Refer to the property files themselves for all documentation on them and how to configure them. After you make all the appropriate changes to your projects property files, remember to save them.

They are straightforward and intuitive. Configuring them is not onerous and if you are “happy” with the default settings, the task should be complete in minutes if not seconds.

Hrmes needs to know which project it is currently running against. This is configured in project.properties file. This file is the only one that needs to live under the core Hrmes project. Copy and paste it into “Hrmes/Framework/conf” folder.

There are few Ant scripts that control the running of the entire framework. All are located under the “AntScripts” directory. Open the Ant view in Eclipse - Window-> Show View -> Ant. Drop all the Hrmes ant scripts (build.xml files) into the Ant Window.

of 66

One configuration file you will almost certainly need to change is the “myBuild.properties” file. Have at look at the default settings and ensure they match the settings on your machines.

Ant scripts are simple to run – just double click them and they will run.

The “Selenium Remote Control Server” script will start the Selenium Server on your machine. After starting the Selenium Server you can now run the Hrmes framework – through the “Start Hrmes tests” Ant script.

Hrmes will run all the tests found under the “Run” folder of your Web Project area.

You now have Selenium IDE tests running on Selenium RC environment, without writing a single line of code. You can not do this with any other framework.

6.2 Test Suites

Creating Hrmes Framework Test Suites

of 66

When you wish to create a series of test scripts designed to be run together simply create a single Hrmes spreadsheet, with a series of ordered worksheets within it.

The framework will run each test script in turn – from left to right. You will also get an individual report generated per worksheet.

6.3 The Hrmes Test Methodology

The Hrmes Framework is designed to be non proscriptive. This flexibility should maximise its usability.Apart from adopting the structure and definition of Selenium’s Fitnesse almost without question – Hrmes will insist on little.

Test Case Design

However, structure and methodology is good for goodness sake. There is an area where Hrmes is prescriptive that might not be immediately obvious.

of 66

Within the Hrmes framework, test scripts should be developed to run independently of all others. There should be no interdependency in regards to either test flow or test data.

This condition may seem an inefficient way to develop test scripts, but it delivers the far bigger prize of scalability. Selenium based tests have to run independently and in parallel in order to scale. If they do not your tests may still be running when you go out of business.

Test authoring in this way will enable greater robustness in the test run. It will also make any test script more readable. Test scripts that jump around fulfilling dependencies, are hard to follow. It is fundamentally poor practice and should be avoided. Make sure all test cases are standalone.

If there is a test flow that is repeated throughout test scripts in a test suite, testers should compose a single test method with all the steps “rolled up”. In this way the overhead is minimised and tests preserve their independence.

Test Case Names

The worksheet names of the Hrmes spreadsheets are synonymous with a test case. Ensure that you give these worksheet names a meaningful and unique description.

If the same test case features in other Hrmes test suites maintain its name. Although, this is not essential you will find that the Hrmes test reports and analysis of results is far clearer on data that has been normalised in this way. It is also far more effective.

of 66

7 Demo - An End to End Example

This section demos the Hrmes Framework by using a real example. It will take you through the recording, exporting, conversion, running and reporting within the framework. This will be a good test of your install and configuration.

We will look at creating the Hrmes test script via the Selenium IDE. Let’s assume you have no test script recorded yet.

7.1 Record

Using the Firefox browser go to the http://www.w3schools.com/ and open the Selenium IDE.

Now record some user activity. Additionally, perform some verification and write a comment for your test after you have finished. You script may look something like the following:

Under the S-IDE options, ensure you have the Hrmes Framework format set, then save you test.

of 66

http://www.w3schools.com/

7.2 Export and ConvertNext export your test in the Hrmes format to the relevant area within your WebProjects directory, e.g. “C:\Hrmes\WebProjects\Demo\SeleniumIDE\Exported”.

of 66

Within Eclipse run the Ant script “Start Hrmes Extraction Process”. You will now have 3 data files with the “Extracted” folder. Remember to refresh the Eclipse view - right click on directory-> refresh. The file we are going to run is the “Demo.xls” file. This is in the Hrmes format.

7.3 Run and ReportDrop the Hrmes file into the Run folder. Ensure that your starting.url is configured to point at your base URL - http://www.w3schools.com/. Now run the “Start Hrmes tests” Ant script.

of 66

After the test has completed you should now have a Hrmes report looking something like the screenshot below. If you do: congratulations as your install and configuration are probably sound. More importantly, your understanding of the Hrmes’s key processes are probably sound too.

If you did not get a report - go back and ensure that you followed all the steps. Also have a look at the troubleshooting guide at the end of this manual.

of 66

8 Using CVS through EclipseThis section assumes you have access to a CVS repository. It is recommended that you implement code control over your project specific Hrmes test assets.

Remember to share your assets

Always consider sharing the assets you have created. There are some that you should not, but amongst those you should are:

• The Hrmes Framework test scripts• The Selenium IDE exported files and original Selenese ones.• The applicationMappings.properties file

If you are familiar with using CVS through Eclipse you may skip the rest of this section.

Sharing your assets …

of 66

Right click on the asset and select Team->Commit. You will get prompted with the screen above.

Enter a comment that will be meaningful to yourself or a third party who may come to use it later. Don't be too concise: "fixed typo" is too short; "fixed typo on test case 10 step 5" or "fixed typo in home page assertions" is OK.

The aim is to make it easier to find the desired change easily from just the commit messages.

Note on Excel files

This point is particularly pertinent when committing spreadsheet data. As Excel files are stored in a binary format, CVS will be unable to provide any other tracking of the changes over time, like it can with ASCII data.

of 66

The converse of this is including too much information. CVS automatically maintains information like the date and time of the commit, the person who made the commit, etc. You don't need to include it in the commit message yourself.

Also remember that Excel can get big – so commit only what you really need to share – do not use CVS as an archive for test reports.

Clicking “Finish” commits the file to the repository and will make it available to the rest of the team and anyone else who comes to work on this project in future.

of 66

9 Understanding the Hrmes Framework Reports

As well as the run-time logs, each time you run the Hrmes Framework it will produce a single Excel spreadsheet report for each worksheet of every Hrmes that is being run. The default location for this is in the “Reports” directory of your WebProjects directory in Eclipse.

This report will match any Hrmes Format file you have produced. Apart from a few extra columns, it will also look much like it. This report will detail the results of your test run and provide the status of each test and related screenshot and stack trace should that test have failed. This is the main report the Hrmes generates.

A test summary report is created for each test run of the Hrmes – including those run through GRID and will look something like the one below.

If your machine is set up to send email, then you will also receive an email notification with this summary report attached. There is a README file under Hrmes->Jars->Ant-libs for further information on getting this setup.

The summary report provides a clear and precise overview of the test run – with number of tests executed, number of passes and fails.

of 66

However, always refer to the aforementioned spreadsheet for detail. It will give you the stack traces and the occasionally priceless: screenshot.

There is also a consolidated report on all theses summaries which provides links back to the data specification being run and the detailed test reports. This can be found under the reporting area, for the demo project this will be:

C:\Hrmes\WebProjects\Demo\Reports\ReportSummary\junit-noframes.html

As well as a general summary on the test runs it contains data on the run-time environment (for both GRID and SRC runs).

Note on why JUNIT reporting via Ant was abandoned

Other JUNIT test frameworks will have dozens of unique test methods per test run and thousands in a test suite. They will also require a team of Java developers to maintain.

The Hrmes Framework is a code super lightweight and has only 1 test method in the entire framework. A report on its test methods will not be informative.

of 66

The only reporting functionality that does remain in this area is the use of the JUNIT report XSL format. This is why the consolidated report is still prefixed with junit.

The Hrmes is still based on the JUNIT test framework.

of 66

10 Developing the Hrmes Framework

It may seem odd having the section on developing the Hrmes Framework after the section on running the Hrmes.

As the Hrmes is an execution-ready framework, you do not have to write any code in order to run it. All the code has already been written.

However, to make full use of all the features of the Hrmes, in particular its ability to perform application mappings and fully extend Selenium it is likely that you will want to develop it.

10.1 The Application mappingsThe Hrmes allows users to abstract out implementation level aspects of their test specifications.

This mapping feature is useful is maintaining a distinction between what is test requirement and what is test implementation. This is often blurred in automation with a detrimental impact on maintenance and clarity. Developing the application mapping layer of the Hrmes is simple. You just create a set of (key value pair) mappings in a property file.

The “applicationMapping.property” file is the place for all these mappings. Please refer to it for all documentation on how to use it.

Mapping application data to test specification data

We are going to go through a simple example to see how application mappings work. Go to: “C:\Hrmes\WebProjects\Demo\test suites\Examples”. Here you will see a set of Hrmes files at various stages of abstraction.

The first Hrmes file we need to look at is the “raw-google” one. This contains the test script as it looks straight after being exported from the S-IDE and converted into the Hrmes format.

of 66

Let’s assume that we wish to use our test data as a test document for our managers and business stake holders. Furthermore, let us assume we are testing a web application that is still under-going development and which has locators that keep changing every release. Lastly let’s assume that we would like to abstract our test data. Using the Hrmes we decide to abstract all the target and value data.

We need to define what these targets will get mapped to. This is done through the applicationMapping.properties file. We already have these mapping defined there as default. The relevant section is extracted below:

of 66

############################################################################ Target/Locator Mappings#query.location=qsearch.button=btnGstarting.url=www.google.co.uk##########################################################################

############################################################################ Value/Data Mappings#query = bbc###############################################################

Mapping test specification data to application data

We have created a set of “key value pairs” that define the map between the application layer and our test requirement. We can now make use of these mappings throughout all our Hrmes test data files.

The end result is a far clearer document. We also have a document which can remain unchanged even when there are changes to the application. We simply change the mapping value rather than all the test specification documents. These documents will change only when the test requirement changes.

In the “Examples” folder we have a template Hrmes file. This can be used to author and structure our tests without having to record through the S-IDE. This is much like the way manual test scripts are initiated.

When to automate

Deciding when to automate at the system test level is more of an art than a science. The main driver tends to be the readiness of the application under test (AUT).

At the system test level, test execution is usually performed manually at the initial stages of a project. The AUT is either not available or too brittle for a machine to test against.

of 66

Hence, tests generally get “migrated” into an automation suite during the life of a project rather than at its inception.

Under circumstances where manual and automated test specifications are distinct, this can be complex and require significant expertise not only to translate the manual test asset, but to get the timing of the migration right. Too early and the automation scripts will need a lot of reworking and significant manual intervention. However, if the tester delays too much, the return on investment from automation gets diminished.

Tests specified at a projects inception, when there is no working web application, will usually require significant reworking as the full functionality of the application gets delivered. However, with a common format, we do not need a new set of test artefacts.

Switching between the two types of testing (manual and automated) is relatively painless. With a common format we do not need to maintain two independent streams of work, i.e. one for manual and one for automation. We have a common mutable document for all types of testing.

10.2 Java Custom MethodsThe Hrmes is a code-free solution to automation. Out of the box, it does not require additional code to run and can be entirely data driven.

However, one of the advanced features of the Hrmes is that its functionality can be extended through Java.

Writing code in certain circumstances will certainly be the most efficient, productive and easiest way to perform certain actions and validations.

Where you have an AUT that has volatile and dynamic test results, requires test flow management, or database validation – then you will want a programmatic solution. So it is vital we have the means to create one.

Writing code for the Hrmes is not particularly difficult as the development framework has already been written and is in place. The most you ever need to write are your custom Java methods.

Under the project specific area you will find the src directory and the ProjectSpeciciMethods Java class file.

Here are examples of project specific Java methods contained in this class that may be of use to you in constructing your own. There are further instructions in the class file itself on what to do with it once you have finished writing your custom methods.

Once in place, all methods will be available for you (and the rest of your team) to invoke via your test specifications in the Hrmes Format.

of 66

For example, let’s say that you wished to combine the Selenium “waitForElement” command with the “verifyText” command. You decide to call your custom method “waitThenVerifyText”.

If you convert the original Selenese format into Java using the S-IDE you end with the following source:

The magic of the Selenium IDE has instantly created a complete Java test class for our 2 commands.

However, using the Hrmes we already have our class and indeed the complete framework to run it in. All we need is the code for our custom method – highlighted in bold.

All custom methods reside in the ProjectSpeciciMethods.java class file. I need to create a custom method called “waitThenVerifyText” within this file. In this case just copy the appropriate code from both Selenium methods above. Then drag and drop the file from “C:\Hrmes\WebProjects\[Project]\src” into the code source of the core Hrmes project – “C:\Hrmes\Framework\src\hrmesFramework”, overwriting the existing one.

of 66

package com.example.tests;

import com.thoughtworks.selenium.*;import java.util.regex.Pattern;

public class NewTest extends SeleneseTestCase {public void setUp() throws Exception {

setUp("http://change-this-to-the-site-you-are-testing/", "*chrome");}public void testNew() throws Exception {

for (int second = 0;; second++) {if (second >= 60) fail("timeout");try { if (selenium.isElementPresent("locator")) break; } catch (Exception

e) {}Thread.sleep(1000);

}

verifyEquals("pattern", selenium.getText("locator"));}

}

As Eclipse, by default automatically compiles projects before they get run there should be no need to explicitly compile.

This method is now available to specify within your Hrmes test specification. With the new custom method available we can now have Hrmes tests that reference it. Like the following:

Unlike many proprietary automation tools, you are not limited to writing methods solely based on the tools API. You can extend the Hrmes by writing any custom code that can be implemented in Java.

So, for example you can write a custom method that interacts with a database to verify the state of backend data at a particular stage in your test. This functionality is invaluable if you need to perform more “whitebox” type of tests. It is one type of test that would be impossible within the Selenium IDE.

of 66

public void waitThenVerifyText(String Locator, String Pattern) {/* * Write code here */

for (int second = 0;; second++) {if (second >= 60) fail("timeout");

try { if (selenium.isElementPresent("locator")) break; } catch (Exception e) {}Thread.sleep(1000);

}

verifyEquals("pattern", selenium.getText("locator"));

}

11 Selenium GRID and the Hrmes Framework

11.1 Selenium GRIDs integration with the Hrmes FrameworkSince version Hrmes_0_1_0_0, the Hrmes Framework is fully integrated with Selenium GRID.

This means that you can evoke any number of instances of the Hrmes to run against the environments you have configured and made available on your Grids’ hub.

The Hrmes provides an Ant script that sets up 6 environments on your default hub location - localhost.

The hub configuration you get with the Hrmes is just to give you a basis to run your Hrmes tests through the Grid. It can all be changed to match your particular

of 66

selenium.grid.server.host.one=localhostselenium.grid.server.environment.one=*chromeselenium.grid.server.port.one=5555selenium.grid.server.args.one=-multiwindow

selenium.grid.server.host.two=localhostselenium.grid.server.environment.two=*chromeselenium.grid.server.port.two=5556selenium.grid.server.args.two=-multiwindow

selenium.grid.server.host.three=localhostselenium.grid.server.environment.three=*chromeselenium.grid.server.port.three=5557selenium.grid.server.args.three=-multiwindow

selenium.grid.server.host.four=localhostselenium.grid.server.environment.four=*iehtaselenium.grid.server.port.four=5558selenium.grid.server.args.four=-multiwindow

selenium.grid.server.host.five=localhostselenium.grid.server.environment.five=*iehtaselenium.grid.server.port.five=5559selenium.grid.server.args.five=-multiwindow

selenium.grid.server.host.six=localhostselenium.grid.server.environment.six=*iehtaselenium.grid.server.port.six=5560selenium.grid.server.args.six=-multiwindow

circumstances. You can have as many environments as you like, configured to support whichever combinations of browser type, version and platform you wish to test for. Assuming, of course you have access to the physical resources to run them on.

If at this point you are wondering what on earth the GRID is, then before diving into the Hrmess implementation of the GRID, refer to the Selenium GRIDs home page on openqa:http://selenium-grid.openqa.org/.

You start the Selenium GRIDs hub and run the Hrmes tests with 2 Ant scripts. One “Start Selenium GRIDs Hub and SRC instances”, will provide the GRID configuration just mentioned.

They can be viewed via the hubs console. By default this is on: http://localhost:4444/console

The second Ant script controls how your Hrmes Framework/SRC instances access these environments. When running instances of SRC (through Hrmes or directly) via the GRID you need to let them know where your hub lives and which environment you want this instance to run against.

of 66

http://localhost:4444/console

http://selenium-grid.openqa.org/

This configuration is stored in the seleniumGRID.properties file and by default has 4 Hrmes Framework/SRC instances defined – something like the abstract above.

You can have as many instances as the hardware your hub references can handle. Each of the 4 instances will run in parallel. Should your hub have available environments you can run any number of configurations. However, you will probably hit hardware issues if you rely too much on your localhost. If your access to hardware is limited, it will start to crawl after about 5 or 6 active Selenium RC instances on the hub.

The key concepts of Selenium GRID are distribution and configuration, rather than just parallelism.

Indeed, running parallel instances of the SRC can be achieved without the GRID. Being able to easily access distributed instances of the SRC running on varying configurations, is the Selenium GRIDs great achievement.

11.2 Selenium GRID and its Hrmes Framework Ant scriptsThe 2 Ant scripts written for the Selenium GRID need to know about a few of the GRIDs’ jar files.

You will need to do this via Eclipse in the similar way you made Ant aware of tools.jar earlier: Eclipse-> Window-> Ant-> RunTime -> CLASSPATH-> Global Entries-> Add Jars.

There are 3 jars you need to add – highlighted in the screenshot below and found under Hrmes/Hrmes Framework/ Selenium GRID.

of 66

selenium.grid.src.environment.one=*chromeselenium.grid.src.starting.url.one=http://www.bbc.co.uk/selenium.grid.src.Hrmes.file.location.one=C:/Hrmes/WebProjects/Demo/Run/Demo-001.xlsselenium.grid.src.environment.two=*chromeselenium.grid.src.starting.url.two=http:// www.bbc.co.uk /selenium.grid.src.Hrmes.file.location.two=C:/Hrmes/WebProjects/Demo/Run/Demo-002.xlsselenium.grid.src.environment.three=*iehtaselenium.grid.src.starting.url.three=http:// www.bbc.co.uk /selenium.grid.src.Hrmes.file.location.three=C:/Hrmes/WebProjects/Demo/Run/Demo-003.xlsselenium.grid.src.environment.four=*iehtaselenium.grid.src.starting.url.four=http:// www.bbc.co.uk /

11.3 Selenium GRID integration with the Hrmes Framework

When the Hrmes Framework runs outside of the GRID it expects to use the standard Selenium RC server. It also expects only one.

It order to make the Hrmes work with the GRID it simply needed to be run with parameters that would tell it which SRC instance to run against. The Hrmes now runs in two modes, one where it will expect multiple instances of SRC and one where it does not.

These modes are controlled by toggling a property in the hrmes.properties file.

In terms of integration that is pretty much it. You start the Hub as you would without the Hrmes and you start the Hrmes instances as you would without the GRID. The

of 66

If you want to use the SRC standard set up leave the selenium.grid.mode.set as false:selenium.grid.mode.set=false

If you want to use the GRID set the selenium.grid.mode.set to true:selenium.grid.mode.set=true

only difference is that you now have multiple instances of the Hrmes on SRC rather than one.

This is important to bear in mind when you are troubleshooting the Hrmes operating with the Selenium GRID. If you have managed to get the GRID working with SRC, i.e. the standard way and if you have also managed to get the Hrmes working without the GRID, you will also get the Hrmes working with the GRID.

The integration of Hrmes Framework with the Selenium GRID is not complex. You do not end up running different “versions” of either the Hrmes or the GRID. Essentially, you are running the GRIDs Hub as you would normally and evoking the Hrmes in a “GRID aware” way.

of 66

12 Release Management and the Hrmes

12.1Upgrading To Newer Versions

The Hrmes Framework has been designed so that its core data is entirely isolated from web project specific data it interacts with.

This makes upgrading from one version to the next straightforward.

When to upgrade

Simply put: if you do not need to upgrade them do not. If you are working happily on a stable release and have no need for new functionality or potential bug fixes stay where you are. All versions of Hrmes are available in perpetuity and you will never be asked to upgrade.However, if you want to make use of new functionality then at some stage you will need to upgrade. Upgrade process

Ensure you backup the file system under Hrmes before you start, just in case you make any mistakes during the upgrade – which does involve deleting files.

Start by deleting the “Framework” directory and all files under it. Make sure that Eclipse has been closed before you begin.

Leave the rest of the Hrmes data, including the “WebProjects” area alone. After the initial install of the Hrmes you never need to check-out that part of the project again. If this area needs changing in a future release – the release notes will tell you how to modify what you have. This part of Hrmes should be under your own code control.

It is a good idea to restart your machine to prevent any locking issues you may encounter.

Before you upgrade, ensure that the hidden .project file for “Framework” is deleted. This is best done in Eclipse. This file is often displayed as a casserole pot in Eclipse.

Once you have deleted the entire “Hrmes” structure, you are ready to get the newer version. You will need to connect to the sourceForge repository to do this. See the notes on the install process should you need reminding on how to do this.

Expand the “Versions” entry in the CVS Repositories tab. Have a look at the versions available by expanding the “Hrmes” entry.

of 66

Right-click on the one that you want and check it out. After the check out process has finished you will have the new version in the same place as the old one was.

12.2The Impact On Shared Data From Upgrades

There are some property files that reside in the web project area that are shared. The core project needs values for these properties, but those values are project related. The starting.url for instance, needs to be defined for the Hrmes to run, but its value is dependent on the particular web project you are testing.

The hrmes.properties, myBuild.properties and seleniumGRID.properties are the three properties files that contain shared data. In rare cases new properties may be added in a new release of the Hrmes. These new properties will be flagged in the release notes of the release that they were introduced in. If your upgrade is not contiguous, remember to check for any new properties from within those intermittent.

Notes on the restructuring the WebProjects area A new release may bring with it new directories and a new layout to the Webproject area under Hrmes. This is rare, but can happen.

of 66

Again changes will be mentioned in the release notes. You can also review any changes in the CVS entries, to assess what this may mean for your project. At the most it may mean creating or renaming directories and moving test assets like properties files and Hrmes test scripts around.

The Hrmes release label number convention

Only major and minor releases, could impact the WebProject area. By convention any changes to the WebProject will mean an increment in the minor release number.

So, if the first 2 digits of the release tag have not changed, the webproject area needs no changes to support a new release. If you do upgrade to a new major or minor release then read the release note and assess the possible impact on your web projects under test.

of 66

13 The Hrmes Frameworks Database SchemaThe Hrmes framework has a MySQL database that may be used for uploading test data – both test cases and test results.

It works with MySQL version 6 and above - as it makes use of a XML interface only available with that version. As MySQL releases are platform specific – this is one application used by the Hrmes than does not come bundled with it. MySQL jars (which are platform independent) do come pre-bundled and are found under the “Jars” directory.

The MySQL download and configuration is simple – see the link below: http://dev.mysql.com/downloads/mysql/6.0.html

As you will want to share and compile data across your web project, set your MySQL instance up on a server, rather than on your local machine.

The Hrmes schema is simple and has been designed solely to make the task of loading XML data into a relational database easy. This works through the MySQL load xml function (on MySQL version 6 or above), which is a robust way of getting XML into a database.

With each run of Hrmes, a set of XML data files are created that correspond to the test cases being run and the results of the test run. These two distinct sets of files reside (by default) under the WebProjects area. Test case XML data gets created under “Data/TestCaseXML” and test run XML data under “Reports”. There are also SQL scripts that reference these XML data files. It is these that are run through the Ant scripts and are used to upload the XML data into the appropriate database tables.

If Hrmes had a roadmap one objective would be to improve the reporting and analysis features of Hrmes. The schema and its functionality will be enhanced in future versions of Hrmes project. Of course, as Hrmes is OSS you do not have to wait until then.

There are Ant scripts to assist data loads and schema build: one for loading Hrmes test data: "Start DB Load of Hrmes Test Case Data", one for test result data: "Start DB Load Of Test Run Data" and one to build the Hrmes schema. All scripts are found under the “AntScripts” directory.

13.1Building the Hrmes SchemaTo make use of this functionality you firstly need to build the Hrmes database schema. Running the Ant script: "Start Hrmes Schema Creation" creates the Hrmes schema objects. This execution the MySQL-schema-creation.sql script under “Hrmes/Framework/ReportDatabase/Schema/”. You need to do this as the root database user. You will also need to create a Hrmes database for this script to succeed. Do this via the MySQL client.

of 66

http://dev.mysql.com/downloads/mysql/6.0.html

13.2 Loading Test Case Data into the Hrmes SchemaAfter each test run, the test case XML data can be loaded into the Hrmes database by clicking the Ant script: "Start DB Load of Hrmes Test Case Data", in the Eclipse Ant window.

All SQL files found in the “Data/TestCaseXML” will be loaded. Delete or archive any you do not want loaded. There are no uniqueness constraints in either of the Hrmes tables, so all data will be inserted regardless.

13.3 Loading Test Results into the Hrmes Schema

In a similar way the test result XML data – which is also created at run time, can be loaded into the Hrmes database by clicking the Ant script: "Start DB Load Of Test Run Data".

Again, all SQL files found in the “Reports” will be loaded.

13.4 Using the SQL Explorer Eclipse Plug-inThis section will take you through the install and configuration of the SQL Explorer plug-in found here:http://eclipsesql.sourceforge.net/

Installing SQL Explorer is entirely optional, as you can of course interact with the Hrmes database using any database tool that you prefer – including those that come with MySQL itself.

However, as Hrmes is Eclipse based you should find the SQL Explorer option the most convenient. See the section earlier on installing Eclipse plug-in, if you need reminding on how this is done. The update site reference is the same link as above.

Once you have installed the plug-in, configure a connection to the Hrmes database. Use the MySQL documentation to help you if this is unfamiliar to you.

Once you have a working connection you can then view data that has been in loaded into the Hrmes database.

You can of course do much more: including running SQL operations against it, creating your own view of the raw data and pretty much anything you can do with similar standalone database applications.

As mentioned SQL Explorer is one of many ways to interact with the Hrmes database, including using the command line.

of 66

http://eclipsesql.sourceforge.net/

13.5Database Run Mode

The advent of a database schema has introduced another way of running Hrmes.

Now you can use test case specifications that reside in the database rather than Hrmes spreadsheets. This can provide the user with much more flexibility over the test data. When run in this way, Hrmes will reference the “Hrmes.raw_test_case_data” table rather than the “Run” directory. All tests will be executed in “load_id” order. This is left NULL initially for the user to configure should they wish to use this mode. They run in the same way as the standard Hrmes tests, but are sourced from the database.

As they run in the same way, you can also run your database test specifications through Selenium GRID.

The GRID configuration is identical to the standard mode. Hrmes simply uses the spreadsheet name of the selenium.grid.src.Hrmes.file.location parameter (within the hrmes.properties file) to match the spreadsheet name in the database – in particular the “raw_test_case_data.spreadsheetname” column .

13.6 Graphical Analysis

The Hrmes project includes the ability to perform graphical analysis on run time and test case data. Although this is fairly limited, it is one area that will be expanded on.

At this stage the inclusion of Entrance with this version is experimental. It is what Entrance calls the “community” or free version and has a few constraints for the Hrmes project. Currently, the ability to run it on the command line is limited with this version – making it hard to run via Ant. It would also be ideal if it was available as an Eclipse plug-in.

That aside, Entrance (http://dbentrance.com/) is on a MySQL platform and can be used to produce some impressive graphics. It is the only application bundled with Hrmes that has to run outside of Eclipse.

Under “Hrmes/Framework/Entrance/charts/benchmark” is the Entrance-SQL code need to produce graphs like the one below.

of 66

http://dbentrance.com/

This graph compares the failure rate of a regression suite with an earlier benchmark run. If there has been no change in the test results we would expect the circles to align with the squares.

The Hrmes scheme provides the raw data of all test runs and test case specifications. It may be used as a basis for a varied graphical representation of your projects test data – both results and specifications.

of 66

14 Debugging tests with the Hrmes Framework

14.1DebuggingUp to now we have generally assumed that your tests run, get reported on and then you move on to the next test cycle.When tests fail you will want to isolate what bits you are most concerned about and reproduce the test within a much more controlled environment than the run-time environment.

Hrmes offers many ways to debug your tests.

14.2Debugging through EclipseAs Hrmes is Eclipse based you can bring all the power of its debugging capabilities to bear to trouble shoot your test.

You can of course debug the any of the code within the Hrmes project, but we will go through the standard case of debugging a test which has failed.

As Hrmes is fundamentally a JUNIT test framework the class we launch the debugger from is the “HrmesTestSuite.java” class file, found under the “Hrmes/Framework->src”. If we right-click on the file within Eclipse we see the option to “Debug as” in the panel second from bottom. Select Java Application will launch the debugger.

As mentioned we can debug any part of the project, but if we are concerned with the workflow a test, we need to locate the part of Hrmes which handles this. This is handled by the “InvocationData.java” class. If we open that file within Eclipse and search for invokeSeleniumMethod, we are in the part of the code that handles the calls to the Selenium API. If we breakpoint here as the commentary suggests we can step through the test workflow step by step.

of 66

Additionally we can put conditions on the breakpoint. We will often want a test to run up to a particular sequence, rather than stop at every command. If we right-click on any breakpoint we set, we select its properties and control under what conditions it breaks.

of 66

In the screenshot above we will stop when we have reached sequence 10.

Controlling Hrmes in this way does take you into territory normally preserved by developers and not all users of Hrmes will feel comfortable with this. There are alternatives.

14.3Debugging through the IDEAs Selenium-IDE versions of every Hrmes tests being run are created at run-time, we can use these as a fallback to the Selenium IDE.

These tests can be found under the WebProjects area, under “SeleniumIDE\Imported”.

14.4Debugging using HrmesWe can add “pause” commands in the Hrmes spreadsheet data to slow down the running of a test at a particular point.

of 66

14.5Debugging ManuallyThere is always a strong element of manual verification of automation test failure. There are many reasons why you will wish to make sure that a failure reported from automation run is actually reproducible by a real user.

One of Hrmes key features is to provide a common format for manual and automated tests. Making use of abstraction you can construct clear test specifications that can be handed over to a test analyst to investigate.

of 66

15 Troubleshooting the Hrmes Framework

15.1Ants CLASSPATH settingsIf you run into any problems running your Ant scripts check your CLASSPATH settings.In Eclipse under Window->Preferences->Runtime->Classpath.

They should look like as they are in the screen shot.

If you are missing these, add them with “Add External JARS”.

15.2 Hrmes Frameworks Java CLASSPATH settingsYour CLASSPATH settings should be as the screenshot below. Under Eclipse, in the package explorer right click on the “Framework” project, then go to properties->Java Build Path. If you are missing Jars then add them through “Add External JARs”.

of 66

Also, if you are using Selenium GRID review that section to ensure you have made Ant aware of the Jar files particular to the GRID.

15.3 Hrmes Frameworks SourceYour “Source” tab should also look something like the one below:

of 66

15.4 The S-IDE Export Process

The Hrmes extraction process is based around exporting native Selenese (HTML) from the Selenium IDE and transforming the data into a spreadsheet format. The Hrmes extraction process works by loading XML data files. After you export from the S-IDE in the Hrmes format, you will have XML data essentially. The file may not have a XML file extension – do not give it one the extraction process renames all files to have this extension before transforming them.

The Hrmes extraction process will enforce format compliance and is fairly strict. This has the advantage of finding any data issues early. However, the exported data needs to be well formed and correctly defined and structured.

The extraction process which builds the Hrmes Excel documents will enforce the Hrmes data structure as defined by the Hrmes XSD - found under ”Hrmes\Framework\ SeleniumIDE \XSD”. If any document exported from S-IDE does not conform they will get rejected. Generally, this will not be an issue as the Hrmes structure is built by simply selecting the Hrmes format option under options in the S-IDE.

However, sometimes data within the S-IDE can get corrupted and the source that gets exported will contain formatting errors. This can happen for many reasons including the switching between S-IDE formats, for instance: HTML to Hrmes.

If your XML files do get rejected then you will need to correct them. Firstly, ensure that you currently have the Hrmes format set – in the Firefox plug-in: S-IDE -> Options -> Format -> Hrmes. Then check the source tab in the IDE. You should see something like the following at the top of your source.

15.5 The Hrmes Framework Source within the Selenium-IDE

of 66

Within the S-IDE click on the source tab to see how your HTML looks in the Hrmes format.

An example of the Hrmes data structure is contained in Hrmes->Framework -> SeleniumIDE ->Formats-> Hrmes-SIDE-Source-Format.txt.

It is critical that the structure in your source matches this. The piece highlighted in bold above will correspond to your Selenium commands and comments that you have created. Ensure that they are situated as above – make any necessary changes in the source itself, save and re-export the data into Hrmes.

15.6S-IDE Export and Extraction Failure

If after exporting your test scripts from the Selenium IDE, you get something like the following “undefined” message in the exported file, the then export from S-IDE has failed:

of 66

<html><head><meta http-equiv="Content-Type" content="text/html; charset=Hrmes-8" /><link rel="selenium.base" href="" /><title>New Test</title></head><body><table cellpadding="1" cellspacing="1" border="1"><thead><tr><td rowspan="1" colspan="3">New Test</td></tr><tr><td rowspan="1" colspan="3">parseToken: clickAndWait</td></tr><tr><td rowspan="1" colspan="3">whenToParse: after</td></tr></thead><tbody><tr>

<td>open</td><td>www.google.com</td><td></td>

</tr></tbody></table></body></html>

undefined<tr><td>open</td><td>/</td><td></td>

</tr><tr>

<td>type</td><td>q</td>

This can be resolved by forcing a reload of the Hrmes format in the Selenium IDE. To do this check the source tab (within the Selenium IDE) as described in the previous section. You will need to re-export your test script.

You may first encounter this problem when you try to extract Selenese data into the Hrmes format. If you do then you will get a null pointer exception and a message in the Eclipse console window like:

“ org.exolab.castor.xml.MarshalException: Content is not allowed in prolog”

of 66

16 Support for the Hrmes Framework

16.1 Use SourceForge to Raise Support Requests

The Hrmes Framework project is an OSS project that hosted on SourceForge.

You can review all issues raised against the project there, via https://sourceforge.net/projects/hrmes/. You will need a sourceforge account.

Once there follow the “Tracker” link under the services drop down.

Raise all requests for support there, including any bugs or indeed any requests for additional features.

It is easy to access and offers standard features for handling these issues.

of 66

https://sourceforge.net/projects/hrmes/

the hrmes universal manual

Documents