quality assurance software test plan · quality assurance software test plan alan greer, chris...
Post on 25-Sep-2020
4 Views
Preview:
TRANSCRIPT
Project Documentation Document PROC-0015
Revision A
Quality Assurance Software Test Plan
Alan Greer, Chris Mayer & Aya Yoshimura
Observatory Sciences Ltd.
July 10, 2015
QA Software Test Plan
PROC-0015, Revision A Page 1 of 88
REVISION SUMMARY:
1. Date: 18 July 2013 Revision: Draft 1 Changes: Original Version
2. Date: 29 August 2014 Revision: Draft 2 Changes: Extensive updates following implementation
3. Date: 27 February 2015 Revision: Draft 3 Changes: Updated for release with Canary 8 including descriptions of configuration
editor
4. Date: 10 July 2015 Revision: A Changes: 3.6 Creating Testing Automation Framework Configuration File section
updated. New section 5.2.7 checkout_directories added. 5.4 Testing Automation Framework Configuration File section updated. Initial formal release.
QA Software Test Plan
PROC-0015, Revision A Page 2 of 88
Table of Contents
1. INTRODUCTION ................................................................................................... 5
1.1 PURPOSE ............................................................................................................... 5 1.2 SCOPE ................................................................................................................... 5 1.3 DEFINITIONS, ACRONYMS AND ABBREVIATIONS ........................................................ 5 1.3.1 Abbreviations ............................................................................................................... 5 2. SYSTEM OVERVIEW ........................................................................................... 6 2.1 INTRODUCTION ....................................................................................................... 6 2.2 IMPLEMENTATION LANGUAGE .................................................................................. 7 2.3 SOURCE CODE ....................................................................................................... 7
3. QUICK START GUIDES ....................................................................................... 8 3.1 INSTALLATION OF THE TEST FRAMEWORK................................................................ 8
3.2 WRITING AND EXECUTING A SYSTEM TEST ............................................................... 8
3.3 WRITING AND EXECUTING A JAVA UNIT TEST ........................................................... 9 3.4 WRITING AND EXECUTING A C++ UNIT TEST ........................................................... 11 3.5 INSTALLATION OF THE TAF AND E2E TEST EXECUTOR ............................................ 12
3.6 CREATING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ...................... 12 3.7 RUNNING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ........................ 13 3.8 CREATING THE ETE CONFIGURATION FILE .............................................................. 14
3.9 RUNNING THE ETE ............................................................................................... 15 4. TESTING ENVIRONMENT ................................................................................. 18
4.1 JAVA TESTS (JUNIT) ............................................................................................. 18 4.1.1 Integration of JUnit ......................................................................................................18 4.2 C++ TESTS (CXXTEST) ......................................................................................... 19 4.2.1 Integration of CxxTest .................................................................................................19 4.3 PYTHON TESTS ..................................................................................................... 19 4.3.1 TestFrameworkStep.py ...............................................................................................20 4.3.2 TestFrameworkStepQA.py ..........................................................................................20 4.3.3 TestFrameworkFailure.py ...........................................................................................24 4.3.4 TestFramework.py .......................................................................................................24 4.3.5 TestFrameworkExecutor.java .....................................................................................24 5. TESTING AUTOMATION FRAMEWORK ........................................................... 25 5.1 INSTALLATION OF THE TESTING AUTOMATION FRAMEWORK ..................................... 25
5.2 TESTING AUTOMATION FRAMEWORK PYTHON CLASSES .......................................... 26 5.2.1 TestingAutomationFramework.py ..............................................................................26 5.2.2 TestingAutomationFrameworkConfigFileReader.py .................................................27 5.2.3 TestingAutomationFrameworkBuilder.py ..................................................................27 5.2.4 TestingAutomationFrameworkExecutor.py ...............................................................28 5.2.5 taf_interactive.py .........................................................................................................28 5.2.6 TAFConfigfileReaderForInteractive.py ......................................................................29 5.2.7 checkout_directories ..................................................................................................29 5.2.8 Report Format ..............................................................................................................29 5.3 RUNNING TESTS USING TAF_INTERACTIVE ............................................................. 31 5.4 TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ..................................... 34 5.4.1 Testing Automation Framework Configuration File Editing Tool.............................37 5.4.1.1 ‘—help’ command........................................................................................................37 5.4.1.2 ‘—list’ command ..........................................................................................................38
QA Software Test Plan
PROC-0015, Revision A Page 3 of 88
5.4.1.3 ‘—new’ command ........................................................................................................39 5.4.1.4 ‘—edit’ command.........................................................................................................42 5.4.1.5 ‘—check’ command .....................................................................................................43 5.4.1.6 Command to edit SITE_CONFIG items ......................................................................44 5.4.1.6.1 –siteconfcopy command ......................................................................................44 5.4.1.6.2 –siteconf command ...............................................................................................45 5.4.1.6.3 – addsiteconf command .......................................................................................45 5.4.1.6.4 – removesiteconf command .................................................................................46 5.4.1.7 –copy command ..........................................................................................................46 5.4.1.8 Interactive configuration tool configure_tests ..........................................................47 6. E2E TEST EXECUTOR ....................................................................................... 51
6.1 END TO END SIMULATOR ...................................................................................... 51 6.2 RUNNING THE ETE ............................................................................................... 51 6.3 CONFIGURATION OF THE ETE ............................................................................... 54 6.3.1 ETE Configuration File ................................................................................................54 6.3.2 Configuration Tool ......................................................................................................56 6.3.2.1 ‘--list’ command ...........................................................................................................56 6.3.2.2 ‘--new’ command .........................................................................................................57 6.3.2.3 ‘--add’ command ..........................................................................................................57 6.3.2.4 ‘--del’ command ...........................................................................................................59 6.3.2.5 ‘--addvm’ command .....................................................................................................59 6.3.2.6 ‘--delvm’ command ......................................................................................................61 6.3.2.7 ‘--list’ command ...........................................................................................................61 6.3.2.8 ‘--edit’ command ..........................................................................................................62 6.3.2.9 ‘--check’ command ......................................................................................................63 6.3.2.10 ‘--copy’ command ..................................................................................................63 6.3.2.11 ‘--help’ command ...................................................................................................63 6.3.2.12 Interactive configure tool ......................................................................................64 6.4 TEST RESULTS AND REPORT GENERATION ............................................................ 67 6.5 CONFIGURING A CRON JOB FOR ETE ...................................................................... 68
7. SETUP OF VMWARE VIRTUAL MACHINE ....................................................... 69 7.1.1 Virtual Machine Snapshot Setup ................................................................................69 7.1.2 Shared Folder ..............................................................................................................69 7.1.3 Remote Control of Virtual Machines ..........................................................................69 8. UNIT TESTING ................................................................................................... 70
8.1 WRITING A UNIT TEST ........................................................................................... 70 8.1.1 Writing a Java Unit Test ..............................................................................................70 8.1.2 Writing a C++ Unit Test ...............................................................................................71 8.2 EXAMPLE JAVA UNIT TEST .................................................................................... 72 8.3 EXAMPLE C++ UNIT TEST ..................................................................................... 74 8.4 ADDING A UNIT TEST TO A TEST SET .................................................................... 77
9. SYSTEM TESTING ............................................................................................. 78 9.1 WRITING A PYTHON SYSTEM TEST ......................................................................... 78 9.2 EXAMPLE PYTHON SYSTEM TEST ........................................................................... 79 9.3 ADDING A PYTHON SYSTEM TEST TO A TEST SET ................................................... 80
10. REGRESSION TESTING .................................................................................... 81 10.1 SETTING UP A REGRESSION TEST RUN .................................................................. 81 11. ETE TEST SERVER............................................................................................ 82
12. FUTURE UPDATES ............................................................................................ 87
QA Software Test Plan
PROC-0015, Revision A Page 4 of 88
13. REFERENCES .................................................................................................... 88
QA Software Test Plan
PROC-0015, Revision A Page 5 of 88
1. INTRODUCTION
1.1 PURPOSE
The intention of this document is to describe the structure of the software and the procedures that form the
Quality Assurance/Quality Control Software Testing for the DKIST CSF, Base, TCS and subsystems
including how this testing is executed, manually or automatically. This document also describes any
setup procedures required and how reports are generated when the tests are executed.
The intended audience of this document is:
The developers of the QA work package, and
The developers of any DKIST systems or subsystems that are required to write tests.
The layout of this document is as follows:
Section 2 provides a brief overview of the entire DKIST test system. Section 3 presents the testing
environment, describing the types of test language (Java, C++ and Python) that are used. Section 4
describes how tests are executed on a machine and the tool available to carry out this execution. Section
5 describes how multiple sets of tests can be run using a tool that coordinates the execution of test sets.
Section 6 describes the setting up of a virtual machine for use within the DKIST test system. Sections 7
and 8 describe the two types of test (Unit and System) and provide descriptions and examples of how to
write each type of test. Section 9 presents a description of a typical regression test setup, and finally
sections 10 and 11 present lists of test candidates for each type of test.
1.2 SCOPE
The software described by this design is the DKIST Quality Control Test System.
The purpose of the DKIST Test System is to provide a suite of software tests, both low and high level,
that thoroughly test all parts of the DKIST CSF and Base code. It also provides a test-framework for
integrating high level subsystem tests into the test framework. All of the individual test tools and
harnesses are integrated into a framework that can be configured to run manually or automatically (or
both) with full reporting features.
1.3 DEFINITIONS, ACRONYMS AND ABBREVIATIONS
Specific definitions, acronyms and abbreviations used in this document are described below. For a more
general glossary and acronym list as used in DKIST see [3].
1.3.1 Abbreviations
CSF Common Services Framework
E2E End-to-end simulator
ETE End-to-end simulator Test Executor
QA Quality Assurance
QC Quality Control
TAF Test Automation Framework
XML Extensible Markup Language
QA Software Test Plan
PROC-0015, Revision A Page 6 of 88
2. SYSTEM OVERVIEW
2.1 INTRODUCTION
The Software QA for DKIST is a framework designed to provide a set of tools for testing the DKIST
CSF, Base, TCS and subsystems. The framework provides tools for QC testing in three languages (Java,
Python and C++) at two levels (unit testing and system testing). The framework also provides all the
necessary tools to perform continuous automated regression testing and is easily configurable for
differing versions of DKIST software. The framework is designed to work on any hardware architecture
by making use of virtual machines but ultimately will execute on the DKIST End-to-end (E2E) simulator
hardware. The framework is extensible to allow future integration of subsystem tests and addition of unit
tests to verify new functionality.
Figure 1. Software Test Framework Overview.
The QA framework is split into three distinct pieces of software.
The first piece of software provides APIs to make writing tests simple and ensures all tests generate the
same style of reports. Tests can be written in Java, C++ or Python as necessary and the APIs provide
methods for verification and reporting test results. A description of the three language types and APIs can
be found in section 4.
The second piece of software is responsible for building specific versions of CSF, Base and other systems
on a machine. It is configured by supplying an XML configuration file which specifies the versions of
each system to checkout, any special build instructions required (e.g. simulated or real hardware) and also
a list of tests to run once the build has completed. This second piece of software is called the Test
Automation Framework (TAF) and is described in more detail in section 5 of this test plan. This Test
Automation Framework can be executed on a real machine or a virtual machine.
The final piece of software is responsible for deploying virtual machines and running test configurations
using the Test Automation Framework described above. This final piece of software will always be
E2E Hardware
Virtual Machine Snapshot
Test Automation Framework
Instance
E2E Test Executor
Virtual Machine Snapshot
Test Automation Framework
Instance
Virtual Machine Snapshot
Test Automation Framework
Instance
QA Software Test Plan
PROC-0015, Revision A Page 7 of 88
running in the background to check when new test sets should be deployed and started. It will run as a
cron job on a machine and there will be a command set to interact with it. The command set will offer
the ability to quickly add a new test set, specify how often the test set should be run and remove test sets
that are no longer required. The final piece of software is called the E2E Test Executor (ETE) and is
described in more detail in section 6.
Together these three software facilities provide a fully automated testing framework that can easily be
extended for new test use cases, additional regression test runs or for trialing new DKIST software
configurations. The use of virtual machines provides a cost saving as no new hardware is required when
an additional test configuration is added to the system. Once configured, the system can run fully
autonomously for as long as is required, with test reports logged and optionally emailed after each test run
completes.
2.2 IMPLEMENTATION LANGUAGE
To cover all types of tests and all CSF code three languages will be used to develop the required testing
framework. The languages are Java, C++ and Python. Java and C++ are required to write tests that
directly interact with corresponding language specific CSF code. Python will be adopted as a higher level
testing suite due to the CSF integrated Jython script executor. Python can also be used to carry out any
high level automation tasks required as part of the software QA.
2.3 SOURCE CODE
All source code written and developed as part of the software QA contract will be provided. The code will
be developed and committed to a separate DKIST QA repository in Tucson.
All source code shall be documented in a manner consistent with good software practices. All source
files shall contain a header containing the version number, revisions, author(s), and functional description.
All functions within a source file shall have a description of the interface and operation of the function,
and all source shall be clearly commented.
QA Software Test Plan
PROC-0015, Revision A Page 8 of 88
3. QUICK START GUIDES
The following sections provide step by step instructions on how to start using the QA framework, without
explaining the details of each step. For a reference guide see the sections later on in this document.
3.1 INSTALLATION OF THE TEST FRAMEWORK
Installing the testing framework into an existing source tree is straightforward. Check out from the "qas"
repository to install the framework in to the tree.
cd $ATST/..
export CVSROOT=:pserver:username@maunder.tuc.noao.edu:2401/home/atst/src/qas
cvs login
cvs checkout atst
cd $ATST
make build_all
The "qas" repository contains all of the framework code required to execute test scripts (Python
unit/system tests, Java unit tests and C++ unit tests) as well as an existing set of CSF and Base test scripts
written to test the CSF code itself.
3.2 WRITING AND EXECUTING A SYSTEM TEST
This section demonstrates how to write and execute an individual system test. It assumes the user already
has a fully built ASDT.
System tests written in python can be placed anywhere and the test framework simply pointed to the
script when executed. Create a new file called helloWorld.py in $ATST and enter the following into the
file.
class Step1( TestFrameworkStepQA ):
def Run( self ):
self.LogMessage("Hello World!")
TestFramework.Register("HELLO.WORLD.0001")
TestFramework.Description("The classic hello world example")
TestFramework.CvsId("$Id:$")
TestFramework.AddStep(Step1())
Save the file and then execute the test.
cd $ATST
./bin/RunSystemTest --script=./helloWorld.py --report=report.txt
As the tests logs a message this will be printed to the screen during the execution of the test.
[2014-07-17 11:04:22.638] (note:HELLO.WORLD.0001.64.1:) ATST.QA.SYSTEM:
[Step1] Hello World!
The test should complete successfully, and once it has the report generated by the test (specified by the
--report option) can be opened to see the results.
[ajg@atst01 atst]$ more report.txt
<?xml version="1.0" ?>
<TEST>
<ID>
QA Software Test Plan
PROC-0015, Revision A Page 9 of 88
HELLO.WORLD.0001
</ID>
<CVSID>
$Id:$
</CVSID>
<VMNAME/>
<SNAPSHOT/>
<TYPE>
system
</TYPE>
<TIME>
2014-07-17 11:04:22.636
</TIME>
<MACHINE>
atst01.observatorysciences.co.uk
</MACHINE>
<RESULT>
PASS
</RESULT>
<FAILURES/>
</TEST>
Looking again at the original test script, creating a test step is simply a case of declaring a class that is a
subclass of TestFrameworkStepQA. The created class must contain the method "def Run(self):"
but otherwise has no limitations. The "Run" method is executed by the test framework. To register the
test step class with the test framework the static method TestFramework.AddStep()is called.
There are many additional helper methods that are available to the test step class (courtesy of the
TestFrameworkStepQA class) and these methods are explained in section 4. For this simple test we made
use of the LogMessage method only.
3.3 WRITING AND EXECUTING A JAVA UNIT TEST
This section demonstrates how to write and execute an individual Java unit test. It assumes the user
already has a fully built ASDT. We will now create a simple unit test that will fail on purpose.
Unit tests written in Java must be placed under the "src/java" folder within the ASDT. Create a new file
called ExampleUnitTest.java in $ATST/src/java/atst/qas/example and enter the following into the file.
package atst.qas.example;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import org.junit.Ignore;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import atst.qas.framework.ITestScript;
@RunWith(JUnit4.class)
public class ExampleUnitTest implements ITestScript
{
@Override
public String getCVSID()
{
return "$Id:$";
}
QA Software Test Plan
PROC-0015, Revision A Page 10 of 88
@Test
public void thisAlwaysPasses() {
}
@Test
public void thisAlwaysFails() {
assertEquals(1.0, 3.0, 0.1);
}
@Test
pubic void thisAlsoAlwyasFails() {
assertEquals(1, 3);
}
}
Java test scripts must be compiled before they can be executed.
make classes
Now the test can be executed
cd $ATST
./bin/RunJavaUnitTest --script=atst.qas.example.ExampleUnitTest --report=report.txt
Notice that a Java test is specified not by the path to the file but instead by the fully qualified class name
after compilation. As the Java unit tests are built using JUnit all of the available features of that test
framework are available when writing the QA unit tests. The main addition to the QA unit tests is the use
of a custom set of runner classes that produce the correct format for the test report file, and the use of the
ITestScript interface which ensures the CVS ID of the test script can be included in the report, which
provides a very useful method for getting back to the correct version of the test script from a test report
file. A full description of the classes used for the Java unit tests can be found in section 4.
The report file generated in this case is shown below.
[ajg@atst01 atst]$ more report.txt
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<TEST>
<ID>ExampleUnitTest</ID>
<CVSID>$Id: ExampleUnitTest.java,v 1.1 2014/02/26 13:34:38 AlanGreer Exp $</CVSID>
<VMNAME/>
<SNAPSHOT/>
<TYPE>unit</TYPE>
<TIME>2014-08-26 08:24:27.897</TIME>
<MACHINE>altair</MACHINE>
<RESULT>FAIL</RESULT>
<FAIL_LOCATION>thisAlwaysFails(atst.qas.example.ExampleUnitTest)</FAIL_LOCATION>
<FAIL_REASON>expected:[1.0] but was:[3.0]</FAIL_REASON>
<FAIL_LOCATION>thisAlsoAlwaysFails(atst.qas.example.ExampleUnitTest)</FAIL_LOCATION>
<FAIL_REASON>expected:[1] but was:[3]</FAIL_REASON>
</TEST>
The output is in the same format as that of the python tests. Note that there is a single <RESULT> entry
showing whether the overall test passed or failed along with separate locations and reasons for the
individual tests. If no entry is present for a specific test then that test has passed.
QA Software Test Plan
PROC-0015, Revision A Page 11 of 88
3.4 WRITING AND EXECUTING A C++ UNIT TEST
This section demonstrates how to write and execute an individual C++ unit test. It assumes the user
already has a fully built ASDT. We will now create a simple unit test that will fail on purpose.
Unit tests written in C++ must be placed under the "src/c++" folder within the ASDT. C++ unit tests
must be built in a special way and so it is necessary to add an additional line to the Makefile.gcc file in
the directory where any C++ unit tests are to be compiled. Edit the Makefile.gcc file in the
$ATST/src/c++/atst/qas/tests/example directory and you will notice the line
include ${ATST}/src/c++/atst/qas/framework/Makefile.gccQAS
This line includes the necessary build rules for compiling C++ unit tests and must be added to any
Makefile.gcc that wants to build tests. Now create a new file called HelloWorld.h in
$ATST/src/c++/atst/qas/tests/example and enter the following into the file. Note the file is a header (.h)
file and not a C++ source file.
#ifndef HELLOWORLD_H_
#define HELLOWORLD_H_
#include "cxxtest/TestSuite.h"
#include <fstream>
#include <iostream>
std::string CVS_ID = "$Id:$";
class HelloWorld : public CxxTest::TestSuite {
public:
void testAlwaysFails(void)
{
// Verify received has been set to true
TS_ASSERT_EQUALS(false, true);
}
};
#endif //HELLOWORLD_H_
It is then necessary to edit the Makefile.gcc file and notify the build system that this test is to be compiled.
Add the following line into the Makefile.gcc:
UNIT_TESTS += HelloWorld
Now make the directory and the test will be compiled. In C++ test files are compiled into executable test
applications that can be executed using the RunC++UnitTest script available from the $ATST/bin
directory.
$ATST/bin/RunC++UnitTest –script=atst/qas/tests/example/HelloWorld --
report=report.txt
A full description of how the C++ unit tests are integrated into the QA system can be found in section 4.
The test output for this test is shown below
QA Software Test Plan
PROC-0015, Revision A Page 12 of 88
<?xml version="1.0" encoding="UTF-8"?>
<TEST>
<ID>HelloWorld.h</ID>
<CVSID>$Id:$</CVSID>
<VMNAME/>
<SNAPSHOT/>
<TYPE>unit</TYPE>
<TIME>2014/08/26:08:38:59.517 GMT</TIME>
<MACHINE>altair</MACHINE>
<RESULT>FAIL</RESULT>
<LOCATION>HelloWorld.h[20]</LOCATION>
<REASON>Error: Expected (false == true), found (false != true)</REASON>
</TEST>
As for the JUnit tests an overall pass/fail is provided by the <RESULT> field along with the location and
reason for each failure.
3.5 INSTALLATION OF THE TAF AND E2E TEST EXECUTOR
The Testing Automation Framework and E2E Test Executor are stored in a separate module in the CVS
repository in Tucson and can be checked out independently from any DKIST software development tree.
export CVSROOT=:pserver:username@maunder.tuc.noao.edu:2401/home/atst/src/qas
cvs login
cvs checkout atst_test
Set the ATST_TEST environment variable to the directory where the atst_test has been checked out.
Build the software as follows.
cd $ATST_TEST
./admin/createDevel --make-all
make install_scripts
After the build the TAF, ETE and configuration file tools will be installed in $ATST_TEST/bin.
The TAF makes use of the python program pexpect. This can be installed with yum.
yum install pexpect
VMWare Workstation is required to run the ETE.
3.6 CREATING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE
A TAF configuration file is required to run the TAF. The configuration editing tools can be used to create
or edit the file. There is a file ($ATST_TEST/src/python/editor/taf_defaults) which stores default values
for the TAF configuration file to be used by the tools. Make sure to change ‘INST_DIR’ value to the
directory where the ATST Software Development Tree is installed. (This is the directory that contains the
ASDT. The TAF sets the ATST environment variable to <INST_DIR>/atst. See 5.4 for more details.) Set
the ‘REPORT’ value to a directory where the reports from the TAF should be kept.
INST_DIR=<Path to the parent directory of ‘atst’>
REPORT=<Existing directory where the TAF reports should be stored>
After editing the taf_defaults file, run configure_taf to create a new TAF configuration file.
QA Software Test Plan
PROC-0015, Revision A Page 13 of 88
cd $ATST_TEST/bin
./configure_taf --new new_taf.xml --id="New TAF configure file"
New TAF configure file (new_taf.xml) created.
In this example we’ll use the atst which is already installed and will run a system test that has been
installed with the Test Framework. Add a test
$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
with index of ‘1’ to the configure file using configure_taf. Also make sure that INITDB and
INSTALL_APP are set to ‘No’.
./configure_taf --edit new_taf.xml --
add_python_system_test='$ATST'/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101
.py --add_python_system_id=1 --initdb=No --install_app=No
TAF config file (new_taf.xml) is saved.
./configure_taf --list new_taf.xml
<<<< new_taf.xml START >>>>
ID:New TAF configure file
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
System tests:
PYTHON id=1 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
Install application:no
Postgresql name:postgresql-9.3
init-db:no
Start ICE service:Yes
<<<< new_taf.xml END >>>>
3.7 RUNNING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE
The TAF is executed as follows.
./TAF --tafconfig=new_taf.xml
After the TAF completes, there should be a report file in the directory specified in the TAF configuration
file.
cd /mnt/hgfs/Reports/2015_02_25_12_16_21/
more SYS_REQ_4.1_0101_PY_20150225121626.xml
<?xml version="1.0" ?>
<TEST>
<ID>
SYS_REQ_4.1_0101
</ID>
<CVSID>
$Id: SYS_REQ_4.1_0101.py,v 1.5 2014/09/10 09:20:43 ChrisMayer Exp $
</CVSID>
<DESCR>
Testing deployment of containers.
</DESCR>
<VMNAME/>
<TAF_ID>
New TAF configure file
</TAF_ID>
<SNAPSHOT/>
QA Software Test Plan
PROC-0015, Revision A Page 14 of 88
<TYPE>
system
</TYPE>
<TIME>
2015-02-25 12:16:31.748
</TIME>
<MACHINE>
qas01
</MACHINE>
<RESULT>
PASS
</RESULT>
<FAILURES/>
</TEST>
If running only one test at a time, then TAF_interactive should be used. This program also requires a TAF
configuration file and executed as follows.
./TAF_interactive new_taf.xml
$ATST is not set. INST_DIR in the TAF configure file is set to /opt.
Use /opt/atst as $ATST? (Y/n) >y
/opt/atst is set to $ATST.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
With this program a test to execute can be specified by its index, and it can also execute tests that are not
listed in the configure file.
3.8 CREATING THE ETE CONFIGURATION FILE
An ETE configuration file is required to run the ETE. The configuration editing tools can be used to
create or edit the file. There is a file ($ATST_TEST/src/python/editor/ete_defaults) which stores default
values for the ETE configuration file to be used by the tools. Setting appropriate values in this file will
make editing an ETE configuration file easier.
In this example, we assume that we have a virtual machine
(/drive/VMWareMachines/ATST_READY/ATST_READY.vmx) with a snapshot (Installed) which was
taken after the atst was installed and made ready to run system tests. We will run the TAF on the virtual
machine with a TAF configuration file (/drive/ATST_TEST/atst_test/bin/new_taf.xml). We also assume
that the virtual machine has a directory (/mnt/hgfs/Reports) for the TAF report files and that is shared to
the host machine’s directory (/drive/ATST_TEST/Reports). Also there is a directory
(/drive/ATST_TEST/FinalReport) to store the final report of the ETE. For this example, the ete_defaults
file can be updated as shown below.
QA Software Test Plan
PROC-0015, Revision A Page 15 of 88
more ../src/python/editor/ete_defaults
VM_FILE=/drive/VMWareMachines/ATST_READY/ATST_READY.vmx
TAF_CONFIG=/drive/ATST_TEST/atst_test/bin/new_taf.xml
REPORT_DIR_PATH_ON_VM=/mnt/hgfs/Reports
REPORT_DIR_PATH_ON_HOST=/drive/ATST_TEST/Reports
FINAL_REPORT_DIR=/drive/ATST_TEST/FinalReport
#EMAIL_ADDRESS=aya@observatorysciences.co.uk
SNAPSHOT=Installed
#SYSTEM_TESTS=all
#JAVA_UNIT_TESTS=all
#CPP_UNIT_TESTS=all
A new ETE configuration file is created as follows.
./configure_ete --new new_ete.xml
Creating a new empty ETE config file...
New ETE config file (new_ete.xml) created.
Please note that the file at this moment is empty. Now add an entry for the test to run at 09:00 UTC.
./configure_ete --add new_ete.xml --time=09:00
Adding a new test instance...
VM order 1 with default values is added.
Default final report directory is used.
ETE config file (new_ete.xml) is saved.
-----ETE configure file - new_ete.xml -----
Time : 09:00
Final report directory: /drive/ATST_TEST/FinalReport
Run order: 1
VM file: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
Snapshot name: Installed
TAF config file: /drive/ATST_TEST/atst_test/bin/new_taf.xml
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
Note that it associated a test instance with a VM instance (we can have multiple instances in a test) using
the values from the ete_defaults file.
3.9 RUNNING THE ETE
The ETE requires –eteconfig parameter to specify the ETE configure file, --vmuser to specify the user of
the virtual machine and –vmpassword to specify the password for the user. More options might be
required depending on what is set in the TAF configuration file (See 6.2). The ETE is intended to be
called by a cron job (see 6.5). A test instance is executed only if the current time is equal to the time that
is specified in the test instance. For example, the test instance created in the previous example wouldn’t
run unless the ETE is called at 09:00 UTC. To run the test instance at any time, we can specify –testtime
option with a desired time. Before executing the ETE, make sure the virtual machine is not running – the
ETE won’t run the test if it is.
./ETE --eteconfig=new_ete.xml --vmuser=atst-qas --vmpassword=<password>
QA Software Test Plan
PROC-0015, Revision A Page 16 of 88
--testtime=09:00
After the ETE completes, there should be a final report file created in the directory specified in the ETE
configuration file as Final report directory.
cd /drive/ATST_TEST/FinalReport/2015_02_25_13_45_41/
more new_ete_final_report.txt
Test Execution: Wed Feb 25 13:45:41 2015
Virtual machine: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
TAF config file: /drive/ATST_TEST/atst_test/bin/new_taf.xml
Results Summary:
Run Pass Fail
Unit 0 0 0
System 1 1 0
Details of failures:
There will also be a TAF report file for the system test in the Report directory.
cd /drive/ATST_TEST/Reports/2015_02_25_13_48_37
more SYS_REQ_4.1_0101_PY_20150225134842.xml
<?xml version="1.0" ?>
<TEST>
<ID>
SYS_REQ_4.1_0101
</ID>
<CVSID>
$Id: SYS_REQ_4.1_0101.py,v 1.5 2014/09/10 09:20:43 ChrisMayer Exp $
</CVSID>
<DESCR>
Testing deployment of containers.
</DESCR>
<VMNAME>
/drive/VMWareMachines/ATST_READY/ATST_READY.vmx
</VMNAME>
<TAF_ID>
New TAF configure file
</TAF_ID>
<SNAPSHOT>
Installed
</SNAPSHOT>
<TYPE>
system
</TYPE>
<TIME>
2015-02-25 13:48:49.345
</TIME>
<MACHINE>
qas01
</MACHINE>
<RESULT>
PASS
</RESULT>
<FAILURES/>
QA Software Test Plan
PROC-0015, Revision A Page 17 of 88
</TEST>
[atst-qas@osllx25 2015_02_25_13_48_37]$
QA Software Test Plan
PROC-0015, Revision A Page 18 of 88
4. TESTING ENVIRONMENT
4.1 JAVA TESTS (JUNIT)
For the Java unit tests the open source test framework JUnit has been selected for use. JUnit is actively
maintained (version 4.11 released November 2012) and has many tutorials as well as an online Javadoc
for its API. The framework can be used to automate the execution of unit tests and is easily extensible.
Tests are written for JUnit by annotating methods with the @org.junit.test annotation. This identifies to
JUnit that this method is a test method. Other annotations are available to simplify the test classes,
including the ability to execute methods before and/or after each test method, and to execute methods
once before and/or after all test methods. Test methods can be ignored if necessary and timeouts for tests
can be set. JUnit also provides other classes that help writing tests, including static assertion methods,
variable checking and comparisons.
4.1.1 Integration of JUnit
It is necessary for the DKIST testing framework to extend the capability of JUnit so that reports can be
generated that adhere to the required report format (see section 5.2.5). To integrate JUnit tests into the
DKIST test framework an additional class (JUnitATSTTestRunner.java) is required that can be executed
from the command line.
$ATST/bin/ajava atst.qas.framework.JUnitATSTTestRunner
This command is executed by the RunJavaUnitTest script introduced in Section 3.3.
The JUnitATSTTestRunner class accepts the following command line parameters:
Parameter Required/Optional Description
script Required Fully qualified test class file to execute.
report Optional Location of the report file that is to be generated for this
test script.
config Optional Not in use.
vmname Optional Name of the virtual machine that this test is running on.
snapshot Optional Name of the snapshot of the virtual machine that this test
is running on.
waitforuser Optional Not in use.
Once executed the class creates the test report file and populates it with the ID, type of test, time of test
execution, machine name that the test is executing on and the versions of CSF and Base software. It then
uses the script command line parameter to obtain the test class file and pass this into the core JUnit test
class, which executes the test and returns the results object. This class parses the results object to obtain
the rest of the information required for the test report. It is the responsibility of the TAF (see section 5) to
provide the required command line parameters for this class.
As well as the JUnitATSTTestRunner class there is also a TestReportGenerator class that is used to
generate the XML report file and an ITestScript interface. All test scripts must implement this interface
so that the test runner class can obtain the CVS version number of the file.
QA Software Test Plan
PROC-0015, Revision A Page 19 of 88
4.2 C++ TESTS (CXXTEST)
The C++ tests will be written to follow as closely as possible the corresponding Java tests for the unit
tests. It is essential to prove the same adherence to requirements in both languages. In cases where it is
not possible to perform the same tests due to the underlying software architecture (e.g. C++ thread model)
then the tests specific to a single language will be clearly marked as such.
For the C++ unit tests the open source test framework CxxTest will be adopted. CxxTest is actively
maintained (version 4.3 released July 2013), has extensive documentation and has already been used for
developing DKIST CSF C++ unit tests in early versions. It is easy to use because it does not require pre-
compiling a library and supports a flexible form of test discovery. It is described as being similar to
JUnit. Using CxxTest involves writing test cases that are defined in C++ header files, which are then
parsed to automatically generate test runners. It is possible to specify the class of runner or a template file
when generating the test files that allow custom code to be executed.
CxxTest provides assertions, fixtures and mocks for test cases, and the resulting compiled CxxTest files
are executables that can be run from the command line.
4.2.1 Integration of CxxTest
It is necessary for the DKIST testing framework to extend the capability of CxxTest so that reports can be
generated that adhere to the required report format (see section 5.2.5). To integrate CxxTest tests into the
DKIST test framework a template file will be used (CxxUnitATSTRunner.h). The CxxUnitATSTRunner
template will define a main entry point for the tests which accepts some additional command line
parameters:
Parameter Required/Optional Description
script Optional Not in use.
report Optional Location of the report file that is to be generated for this
test script.
vmname Optional Name of the virtual machine that this test is running on.
snapshot Optional Name of the snapshot of the virtual machine that this
test is running on.
Once executed the template main creates the test report file and populates it with the ID, type of test, time
of test execution, machine name that the test is executing on and the versions of CSF and Base software.
It then executes the test listener which will be a custom class to ensure the rest of the test data is appended
to the report in the correct format. It is the responsibility of the TAF (see section 5) to provide the
required command line parameters for this class.
4.3 PYTHON TESTS
The test-framework is a set of Python classes that provide an easy to use API for writing system tests.
The classes will be adapted from the current set of test-framework classes used for the TCS acceptance
tests. The classes are executed using the available CSF Jython interpreter and so have full access to the
Java CSF services and public API. This provides the ideal method for system tests that can interact with
CSF applications through the same set of public interfaces as other CSF applications (including the JES).
The test classes simply wrap the available CSF API with additional handlers to automatically catch
exceptions and ensure test data is logged as expected. The test-framework is executed from within a CSF
Java VM and so one additional Java class is required to start the required CSF standalone application.
This is explained in more detail below.
QA Software Test Plan
PROC-0015, Revision A Page 20 of 88
The test-framework provides the ability to log messages throughout the test execution and these messages
are all logged to files. The test-framework also logs a report in the format required by the TAF (see
section 5.2.5).
The following sections list the classes that form the test-framework and provide a description of each
class.
4.3.1 TestFrameworkStep.py
This is a Python class. This forms the base class for developers of test-framework scripts. It contains a
small set of methods, the minimum required set for writing a test script. The following API is provided
by this class:
Method Description
AddFailure Parameter:reason - The reason for the failure
Parameter:step - The test step
Parameter: loc - The location of the failure.
This method is internal to the test framework and should not be called or
overridden. When a test fails the location and reason for the failure is added as
information by calling this method.
LogMessage Parameter: message - The message to log.
Logs a message using the CSF Log service and also to the test data file. The
message is prefixed with the unique test ID and step for future identification.
Execute Parameter: test
Parameter: step
Parameter: data
Parameter: failures
Parameter: logdb
Parameter: test_data
This method is internal to the test framework and should not be called or
overridden. The execute method is called by the test framework for each test
step and manages the calls to the "Run" and "Cleanup" methods (see below). It
also is responsible for catching exceptions that are generated by the test scripts.
Run This method is empty and it is expected to be overridden by the individual test
step classes. It is called by the "Execute" method when the test is executed. If
this is not overridden then it raises a NotImplementedError.
Cleanup This method is empty and it can be overridden by the individual test step
classes. It is called by the "Execute" method after the test step has finished.
This method will always be called, whether the test has passed or not and is
intended to give test authors the potential to clean up resources, even if a test has
failed or some other unexpected exception has occurred.
4.3.2 TestFrameworkStepQA.py
This is a Python class sub classed from TestFrameworkStep. Developers of system test-framework
scripts will subclass this and make use of the helper methods provided. This class provides hooks into all
of the CSF services. It also provides helper methods for carrying out certain tasks as well as the ability to
check for standard CSF responses and error messages. By using the helper methods the correct format of
QA Software Test Plan
PROC-0015, Revision A Page 21 of 88
reports are generated for the tests which can be parsed by the TAF. The following public API is provided
by this class:
Method Description
Fail Parameter: message - A message to log as the reason for the failure.
Parameter: loc - An optional parameter that specifies where the test has failed.
This forces the test step to fail and stop any further execution. It will result in an
instance of the TestFrameworkFailure class being thrown. A message is passed
to this class to explain why the test has failed.
AllFail Parameter: message - A message to log as the reason for the failure.
Parameter: loc - An optional parameter that specifies where the test has failed.
This forces the test step to fail and stop any further execution. Unlike Fail
however it also cancels any subsequent steps. It results in an instance of the
TestFrameworkFailure class being thrown. A message is passed to this class to
explain why the test has failed.
Store Parameter: attribute - An DKIST attribute to store that is available to all further
test steps that are to be executed.
The test-framework has access to all CSF services, including the CSF cache.
This method allows test steps to store values into the cache for future use. These
values are preserved across test steps.
Retrieve Parameter: name - The name of an DKIST attribute that was stored previously.
The test-framework has access to all CSF services, including the CSF cache.
This method allows test steps to retrieve previously stored values from the
cache.
StoreObject Parameter: name - A name that can be used to reference the object
Parameter: o - an arbitrary object that can be accessed by later test steps
This allows any test data that is required by a test to be stored and then retrieved
by name by later test steps
RetrieveObject Parameter: name - The name of an object that was stored previously
Look up and return test data that has been stored by a previous test step
AsyncSubmit Parameter: controller - The name of the controller to submit to.
Parameter: configuration - An DKIST configuration to submit to the controller.
Submit a configuration to a controller and return immediately with the response
code.
AsyncDelayedSubmit Parameter: controller - The name of the controller to submit to.
Parameter: configuration - An DKIST configuration to submit to the controller.
Parameter;delay - Delay in seconds
Submit a configuration to a controller after a specified delay and then return
immediately with the response code.
SyncSubmit Parameter: controller - The name of the controller to submit to.
Parameter: configuration - An DKIST configuration to submit to the controller.
Submit a configuration to a controller and wait until the controller reports that it
QA Software Test Plan
PROC-0015, Revision A Page 22 of 88
is complete before returning with the response code and any error code.
ParseResponse Parameter: response - The response code to parse.
Parse a response code into a string message.
GetErrorMessage Parameter: configurationID - The ID of the submitted configuration.
If a submit failed then query the error message string returned from the
controller.
N.B. Not yet fully implemented
SetValues Parameter: name - The name of the component/controller to set.
Parameter: table - An DKIST attribute table to send to the component/controller
with attributes containing values to set.
Connect to a component/controller and issue a set.
GetValues Parameter: name - The name of the component/controller to get from.
Parameter: table - An DKIST attribute table containing empty attributes with the
names of values that should be retrieved.
Connect to a component/controller and issue a get, returning an attribute table.
GetValue Parameter: cname - The name of the component/controller to get from.
Parameter: aname - The name of the attribute for the value that should be
retrieved.
Connect to a component/controller and issue a get for a single attribute returning
the value as a string
GetHealth Parameter: name - The name of the component/controller to retrieve the health
from.
Retrieve the current health of a component/controller as a string i.e. GOOD,
ILL, BAD etc.
GetLifecycle Parameter: name - The name of the component/controller to retrieve the
lifecycle from.
Retrieve the current lifecycle state of a component/controller as a string
Post Parameter: name - The name of the event channel to post on.
Parameter: table - An DKIST attribute table containing the data to post.
Post an event immediately.
AsyncPost Parameter: name - The name of the event channel to post on.
Parameter: table - An DKIST attribute table containing the data to post.
Parameter: delay - The time in seconds to delay before posting the event.
Post an event in the future. Processing continues which allows the test time to
listen for the same event.
Listen Parameter: event - The name of the event channel to listen to.
Parameter: items - A list of specific attributes to listen for.
Parameter: wait - A maximum time in seconds to wait for the event. Default is
5.0 seconds.
Listen for a specific event and return an attribute table containing the items
QA Software Test Plan
PROC-0015, Revision A Page 23 of 88
requested
ListenWithFilter Parameter: event - The name of the event channel to listen to.
Parameter: items - A list of specific attributes to listen for.
Parameter:filter - A list of rules. For example, [["__.name","CMP2"],
["category","alarm"]]
Parameter: wait - A maximum time in seconds to wait for the event. Default is
5.0 seconds.
Listen for a specific event with specific contents and return an attribute table
Alarm Parameter: atype - The type of alarm to raise.
Parameter: reason - The reason for raising the alarm.
Raise an alarm.
ListenForAlarm Parameter: cname - The name of the controller that the alarm originates from.
Parameter: atype - The type of the alarm to listen for.
Parameter: wait - The maximum time in seconds to wait for the alarm. Default
is 5.0 seconds.
Listen for a specific alarm to be raised.
Uninit Parameter: name - The name of the component/controller to uninit.
Helper method for un-initializing a component or controller.
VerifyLoaded Parameter: name - The name of the component/controller to verify.
Parameter: timeout - The timeout in seconds. This parameter is optional. If none
is supplied then the default is 180
Return: status - Set to 0 if successful. Non-zero otherwise
Check the lifecycle state of the specified component/controller and if necessary
change the lifecycle to "loaded".
VerifyInitialized Parameter: name - The name of the component/controller to verify.
Parameter: timeout - The timeout in seconds. This parameter is optional. If none
is supplied then the default is 180
Return: status - Set to 0 if successful. Non-zero otherwise
Check the lifecycle state of the specified component/controller and if necessary
change the lifecycle to "initialized".
VerifyRunning Parameter: name - The name of the component/controller to verify.
Parameter: timeout - The timeout in seconds. This parameter is optional. If none
is supplied then the default is 180
Return: status - Set to 0 if successful. Non-zero otherwise
Check the lifecycle state of the specified component/controller and if necessary
change the lifecycle to "running".
SearchLog Parameter: whereclauses - A list of search parameters used to query the
database. It is a list of tuples of three items. For example:
[["category","=","HEALTH"], ["time_stamp",">","2014-02-24 16:00:00.000"]]
All of these will be concatenated with AND
QA Software Test Plan
PROC-0015, Revision A Page 24 of 88
This is a helper method for checking log messages that have been stored in the
log database. It allows any search string to be specified and returns a list of
objects that contain all of the log entries that satisfy the search criteria.
LoadAppData Parameter: csvfile - name of file to load
Uses AppReader to load an application database file
4.3.3 TestFrameworkFailure.py
This is a Python class. This is an internal class used by the test-framework to check for failures that are
caused by unexpected problems with the test scripts. Developers of test scripts will check for pass or fail
criteria but sometimes the test script itself might fail, due to generated exceptions or unexpected data.
This class is used when catching those types of problems.
4.3.4 TestFramework.py
This is a Python class. This class is the main python class that derivatives of TestFrameworkStep.py can
register with. This class stores data relating to the individual steps, such as the time of execution and
unique ID, and executes the tests recording the output to the specified report files. This class will handle
any types of failure (expected failure or unexpected test failure) and ensure that the test-framework is able
to continue.
4.3.5 TestFrameworkExecutor.java
This is a Java class. This class will provide the command line interface for running tests using the test-
framework. The class accepts parameters which point it to a test script file and the location of test reports
and test data files. It starts up the Java CSF services using the Standalone API with a full set of service
tools, and then creates an instance of the Jython executor. Once the Jython executor is ready the python
script is loaded and executed. The test report location is passed to the TestFramework python class which
handles writing the report and test data files.
QA Software Test Plan
PROC-0015, Revision A Page 25 of 88
5. TESTING AUTOMATION FRAMEWORK
The Testing Automation Framework (TAF) is responsible for coordinating a large set of tests. It consists
of a set of Python scripts, Java unit tests and CPP unit tests that are executed on the (virtual) machine
where the tests are to be run. The TAF will run under the assumption that tests are to be executed from a
clean checkout. It will check its configuration file to find out which version or versions of the CSF, Base
and subsystems should be checked out. The TAF will then check out the versions into the designated
directory and perform all installation steps required to have a fully operational CSF installation.
Once the installation is complete the TAF will run in sequence all of the pre-selected system tests,
followed by the unit test. The tests that execute on a particular instance of the TAF are entirely
configurable, and as multiple TAF instances can run simultaneously within separate virtual machines it is
conceivable that several versions of CSF can be regression tested at the same time (subject to the physical
memory constraints of the underlying hardware).
It is possible to configure an instance of the TAF to not execute any tests at all, simply checkout the
required version of CSF, Base and the subsystems. This would be useful for a situation where tests
executing under another TAF instance on a separate virtual machine expect to connect to remote
application instances. A set of virtual machines can be configured to mimic a fully networked system
ready for network tests. This of course would not be suitable for testing the network architecture, only for
testing the software behavior within a distributed deployment.
The TAF is responsible for collecting all of the reports from the tests and producing the final report which
will be saved to a specified location and emailed to a specified list of addresses. The TAF is not
responsible for sending the emails containing the reports; that is the responsibility of the ETE.
5.1 INSTALLATION OF THE TESTING AUTOMATION FRAMEWORK
The testing automation framework (TAF) is stored a separate module in the CVS repository in Tucson.
As such it can be checked out independently from any DKIST software development tree (ASDT). The
instructions below assume that the software is to be checked out into a directory $ATST_TEST.
cd $ATST_TEST/..
export CVSROOT=:pserver:username@maunder.tuc.noao.edu:2401/home/atst/src/qas
cvs login
cvs checkout atst_test
Once checked out the software can be built as follows
cd $ATST_TEST
./admin/createDevel --make-all
make install_scripts
After the build the TAF script will be installed in $ATST_TEST/bin.
Note that the TAF makes use of the python program pexpect. This can be installed with yum.
yum install pexpect
The TAF is executed as follows:
export ATST_TEST=<path to ATST TEST installation>
$ATST_TEST/bin/TAF --tafconfig=<TAF configuration file>
QA Software Test Plan
PROC-0015, Revision A Page 26 of 88
The TAF configuration file is an xml file. Its format is described in Section 5.3. The TAF script and
underlying classes can be passed a number of options but only the tafconfig option is compulsory. Other
options are described in Section 5.2.1 below.
The tests in the TAF configuration file can be executed interactively using TAF_interactive. See section
5.3 for more details.
[atst-qas@osllx25 bin]$ ./TAF_interactive TAF.xml
$ATST is not set. INST_DIR in the TAF configure file is set to /opt.
Use /opt/atst as $ATST? (Y/n) >
/opt/atst is set to $ATST.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
5.2 TESTING AUTOMATION FRAMEWORK PYTHON CLASSES
5.2.1 TestingAutomationFramework.py
This class provides the entry point for the TAF and parses the command line parameters. It coordinates
and monitors the other classes to ensure the checkout, build and test execution is executed, and it ensures
that if something goes wrong the test sequence doesn't halt prematurely. The only compulsory command
line parameter is the location of the TAF configuration file. This file is then parsed by the class
TestingAutomationFrameworkConfigFileReader described below.
The options accepted by this class are as follows:
Parameter Required/Optional Description
tafconfig Required The TAF config file and path
cvsuser Optional Set the CVS user if it is required to check out an ATST
module rather than use a currently checked out location
cvspass Optional Set the CVS password associated with the given CVS
user
vm Optional Set the virtual machine name to be recorded in the test
report
snapshot Optional Set the snapshot name to be recorded in the test report
waitforuser Optional Specify this option to run tests step by step. TAF will wait
for user’s input before running a test and before each test
step.
systemtests Optional Specify Ids of the system tests to run, separated by ‘,’.
‘all’ can be set to run all the tests.
Steps of each test to be executed can also be specified
inside []. For example to execute a test (id=1), steps from
1 to 10, this option should be set to –systemtests=1[1-10].
QA Software Test Plan
PROC-0015, Revision A Page 27 of 88
N.B. If any of the options systemtests, javaunittests and
cppunittests is specified, then TAF will execute only the
tests that are specified. If none of these three options is
specified, then TAF will execute all the tests listed in the
configure file.
javaunittests Optional Specify Ids of the Java unit tests to run, separated by ‘,’.
‘all’ can be set to run all the tests.
cppunittests Optional Specify Ids of the Java unit tests to run, separated by ‘,’.
‘all’ can be set to run all the tests.
help Optional Display help on configuration options. The script will
then exit
5.2.2 TestingAutomationFrameworkConfigFileReader.py
The TestingAutomationFrameworkConfigFileReader class is responsible for parsing the TAF
configuration file and executing the corresponding build and execution procedures according to that file.
The class will search through all of the versions of software packages defined and invoke the builder class
below to first checkout all of the software, and then build and install it. Once this has completed it will
parse the configuration file for test script definitions and execute those using the executor class described
in Section 5.2.4.
The full list of tags that can be present in the configuration file can be found in Section 5.3.
5.2.3 TestingAutomationFrameworkBuilder.py
This class is responsible for checking out, configuring and building the specified versions of DKIST
software. The class provides helper methods for these checkout and compilation tasks. The table below
describes the public methods.
Method Description
checkout Parameter repofolder: The directory containing the CVS repository
Parameter modulname: The name of the module to be checked out
Parameter version: The version of the module to be checked out
Checks out a software module using cvs.
configsite Parameter site_config_temp: The site config template file
Parameter site_config: The site configuration file
Parameter configlist: List of config file variables and values
Sets specific variables within the site.config file.
createDevel Parameter param: Option to createDevel script
Executes the createDevel administration script with teh supplied option.
Currently accepted values are --init-bin, --init-data, --init-doc, --init-src, --
init-tools, --make-all
initdb Helper method, calls createDevel with --init-db.
checkPostgresqlService Parameter posgresql_name: Name of postgreqql service
Checks the status of the postgresql service using the name supplied in order to
check if it is runnin
startIceServices Checks if ICE services are running
QA Software Test Plan
PROC-0015, Revision A Page 28 of 88
make Parameter param: Specifies what is to be made
Performs a make command. The method accepts a variable which is passed to
the make command, specifying what should be built.
5.2.4 TestingAutomationFrameworkExecutor.py
This class is responsible for executing tests. It provides three methods to make test execution simple to
setup.
Method Description
runJavaUnitTest Parameter install_loc: Location of the DKIST development tree
Parameter test; Name of Java test class
Parameter result: Name of output test report file
Parameter config: Name of configuration
Parameter vmname: Name of virtual machine
Parameter snapshotname: Name of snapshot
Parameter waitforuser: ( not used)
Parameter tafid: ID of the configuration
The method executes a Java unit test
runCPPUnitTest Parameter install_loc: Location of the DKIST development tree
Parameter test; Name of C++ test executable
Parameter result: Name of output test report file
Parameter config: Name of configuration
Parameter vmname: Name of virtual machine
Parameter snapshotname: Name of snapshot
Parameter waitforuser: Whether to wait for user input (not used)
Parameter tafid: ID of the configuration
The method executes a C++ unit test
runSystemTest Parameter install_loc: Location of the DKIST development tree
Parameter test; Name of jython test file
Parameter result: Name of output test report file
Parameter config: Name of configuration
Parameter vmname: Name of virtual machine
Parameter snapshotname: Name of snapshot
Parameter waitforuser: Whether to wait for user input
Parameter tafid: ID of the configuration
Parameter steps: Steps to run
The method executes a system test
5.2.5 taf_interactive.py
This is a program to execute tests listed in TAF configure file interactively. To run this program, specify a
TAF configuration file. When this program starts up the configuration file is parsed by
TAFConfigfileReaderForInteractive.py and obtains a list of the tests specified in the file. This program
has a list of command options to run tests interactively. See Section 5.3.
QA Software Test Plan
PROC-0015, Revision A Page 29 of 88
5.2.6 TAFConfigfileReaderForInteractive.py
The TAFConfigfileReaderForInteractive class is responsible for parsing the TAF configuration file and
obtaining test information for the taf_interactive.py.
5.2.7 checkout_directories
The checkout_directories entry specifies the list of directories to checkout for each module. The TAF
reads this file to decide which directories to checkout. Currently the file is set as follows:
DEFAULT{
atst/resources/screens/$MODULE
atst/resources/images/$MODULE
atst/resources/$MODULE
atst/src/java/atst/$MODULE
atst/src/c++/atst/$MODULE
atst/src/idl/atst/$MODULE
atst/src/jython/atst/$MODULE
atst/doc/guides/$MODULE
atst/doc/uml/$MODULE
atst/admin/$MODULE
atst/tools/$MODULE
}
osl
{
atst/src/c++/$MODULE
}
The format is <module name>{list of directories (one in each line)}. The ‘DEFAULT’ entry is a special
entry – it is used as a default set of directories to checkout when the TAF checks out a module. The
$MODULE text will be replaced with the module name. When there is an entry of a specific module – in
this case ‘osl’ – the TAF checks out only the directories specified in the entry for the module. In this
example, for the ‘osl’ module, it will only checkout ‘atst/src/c++/osl’. With this file it is possible to
change the list of directories to checkout without changing the underlying python source code.
5.2.8 Report Format
Every test that is executed under the TAF must produce a report in the correct format so that it can be
parsed and added to the global report. The format required is xml. Below is a description of the elements
that can form the report.
Element Required Multiple Description
<test> Yes No Parent element, contains all other test report elements.
<id> Yes No Unique identifier for the test.
<type> Yes No Unit or system test.
<time> Yes No Timestamp for when the test was executed.
<machine> Yes No Name of the machine that the test was executed on.
<version> No Yes Version information for the test. This can include
versions of any of the CSF software or subsystems.
<result> Yes No This should be set either to SUCCESS or FAIL.
<fail_location> No Yes This contains information relating to where the test
failed. Multiple entries are possible to allow a
QA Software Test Plan
PROC-0015, Revision A Page 30 of 88
complete description of the location if available.
<fail_reason> No Yes The reason for failure of the test.
An example report is shown below. In this example the test has failed.
<?xml version="1.0" ?>
<TEST>
<ID>
SYS_REQ_4.1_0550
</ID>
<CVSID>
$Id: SYS_REQ_4.1_0550_2.py,v 1.1 2014/03/24 15:44:29 AlanGreer Exp $
</CVSID>
<DESCR>
Verify the controller action queue satisfies requirement 4.1-0550.
</DESCR>
<VMNAME>
/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx
</VMNAME>
<TAF_ID>
HEAD version install and Test
</TAF_ID>
<SNAPSHOT>
ETEready
</SNAPSHOT>
<TYPE>
system
</TYPE>
<TIME>
2015-02-15 10:05:29.004
</TIME>
<MACHINE>
qas01
</MACHINE>
<VERSION name="CSF">
HEAD
</VERSION>
<VERSION name="Base">
HEAD
</VERSION>
<RESULT>
FAIL
</RESULT>
<FAILURES>
<FAILURE>
<STEP>
7
</STEP>
<REASON>
Unexpected order of execution, see log messages
</REASON>
<LOCATION>
StepRunActionQueueTests
</LOCATION>
</FAILURE>
</FAILURES>
QA Software Test Plan
PROC-0015, Revision A Page 31 of 88
</TEST>
This report format can easily be parsed by an xml parser in any language. It does not have to contain all
of the tags and can repeat tags if necessary, which makes it very flexible for describing the outcome of
tests. Multiple reports can simply be added together and then mined for specific tags.
5.3 RUNNING TESTS USING TAF_INTERACTIVE
The program TAF_interactive can be used to run tests listed in a TAF configure file interactively.
Usage: ./TAF_interactive <TAF configure file path>.
[atst-qas@osllx25 bin]$ ./TAF_interactive TAF.xml
$ATST is not set. INST_DIR in the TAF configure file is set to /opt.
Use /opt/atst as $ATST? (Y/n) >
/opt/atst is set to $ATST.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
The program reads the TAF configure file and uses <INST_DIR> to set $ATST environment variable.
The program reads the <TESTS> section of the configure file and adds them to its internal list if the
‘index’ attribute is set to the test entry. It is also possible to run a test not listed in the TAF configuration
file by specifying the test file path name or class name.
Below is an example of listing JAVA unit tests.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
>5
16 Java unit tests
1:atst.qas.tests.csf.REG_CANARY_7_ATCS_627
2:atst.qas.tests.csf.REG_CANARY_7_ATCS_632
3:atst.qas.tests.csf.REG_CANARY_7_ATCS_636
4:atst.qas.tests.csf.REG_CANARY_7_ATCS_639
QA Software Test Plan
PROC-0015, Revision A Page 32 of 88
5:atst.qas.tests.csf.REG_CANARY_7_ATCS_642
6:atst.qas.tests.csf.REG_CANARY_7_ATCS_659
7:atst.qas.tests.csf.REG_CANARY_7_ATCS_666
8:atst.qas.tests.csf.REG_CANARY_7_ATCS_692
9:atst.qas.tests.csf.REG_CANARY_7_ATCS_695
10:atst.qas.tests.csf.REG_CANARY_8_ATCS_696
11:atst.qas.tests.csf.REG_CANARY_8_ATCS_790
12:atst.qas.tests.csf.REG_CANARY_8_ATCS_791
13:atst.qas.tests.csf.REG_CANARY_8_ATCS_792
14:atst.qas.tests.csf.REG_CANARY_8_ATCS_801
15:atst.qas.tests.csf.REG_CANARY_8_ATCS_822
16:atst.qas.tests.base.REG_CANARY_7_BASE_135
This is an example of running the Java unit test at index 1.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
>2
Id? >1
Running Java unit test atst.qas.tests.csf.REG_CANARY_7_ATCS_627
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<TEST>
<DESCR/>
<ID>REG_CANARY_7_ATCS_627</ID>
<CVSID>$Id: REG_CANARY_7_ATCS_627.java,v 1.1 2014/03/12 16:43:45 AlanGreer
Exp $</CVSID>
<VMNAME/>
<TAF_ID>New TAF</TAF_ID>
<SNAPSHOT/>
<TYPE>unit</TYPE>
<TIME>2015-02-16 17:39:55.816</TIME>
<MACHINE>qas01</MACHINE>
<RESULT>PASS</RESULT>
</TEST>
This is an example of running system test by specifying the test script file’s path.
======================= menu ======================
1:Run system tests from a list.
2:Run Java unit tests from a list.
3:Run CPP unit tests from a list.
4:List system tests.
5:List Java unit tests.
6:List CPP unit tests.
7:Run system test (by test file path).
8:Run Java unit test (by class name).
QA Software Test Plan
PROC-0015, Revision A Page 33 of 88
9:Run CPP unit test (by class path).
99:Exit this program.
===================================================
>7
Python system test > $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0300.py
Steps (default:all) >
Running Python system test
$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0300.py
[2015-02-17 09:48:13.832] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step1] Loading a java container SYSREQ410300JAVA1 onto a computer
[2015-02-17 09:48:16.884] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step2] Verifying component is registered
[2015-02-17 09:48:16.911] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step2] The container SYSREQ410300JAVA1 is registered.
[2015-02-17 09:48:16.913] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step3] Loading a java container SYSREQ410300JAVA1 onto a computer
[2015-02-17 09:48:19.919] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step4] Starting up SYSREQ410300JAVA1
[2015-02-17 09:48:21.968] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step5] Adding controller SYS.410300.CTRL.1
[2015-02-17 09:48:24.140] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step5] Controller SYS.410300.CTRL.1 loaded
[2015-02-17 09:48:24.145] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step6] Initializing SYS.410300.CTRL.1
[2015-02-17 09:48:26.204] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step7] Starting up SYS.410300.CTRL.1
[2015-02-17 09:48:30.214] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step8] Verify the language for this controller is JAVA.
[2015-02-17 09:48:30.219] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step8] Controller reports its language as: JAVA.
[2015-02-17 09:48:30.222] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step9] Shutting down SYSREQ410300JAVA1
[2015-02-17 09:48:36.984] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step10] Loading a c++ container SYSREQ410300CPP1 onto a computer
[2015-02-17 09:48:39.992] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step11] Verifying component is registered
[2015-02-17 09:48:40.005] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step11] The container SYSREQ410300CPP1 is registered.
[2015-02-17 09:48:40.007] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step12] Starting up SYSREQ410300CPP1
[2015-02-17 09:48:42.020] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step13] Adding controller SYS.410300.CTRL.2
[2015-02-17 09:48:44.031] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step13] Controller SYS.410300.CTRL.2 loaded
[2015-02-17 09:48:44.034] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step14] Initializing SYS.410300.CTRL.2
[2015-02-17 09:48:46.055] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step15] Starting up SYS.410300.CTRL.2
[2015-02-17 09:48:50.062] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step16] Verify the language for this controller is C++.
[2015-02-17 09:48:50.066] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step16] Controller reports its language as: C++.
[2015-02-17 09:48:50.070] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:
[Step17] Shutting down SYSREQ410300CPP1
-----------------------
QA Software Test Plan
PROC-0015, Revision A Page 34 of 88
Test report
-----------------------
<?xml version="1.0" ?>
<TEST>
<ID>
SYS_REQ_4.1_0300
</ID>
<CVSID>
$Id: SYS_REQ_4.1_0300.py,v 1.2 2014/09/12 08:00:13 ChrisMayer Exp $
</CVSID>
<DESCR>
Verify that controllers can be implementable in either Java v1.7 or in
C++11 as specified in requirement 4.1-0300.
</DESCR>
<VMNAME/>
<TAF_ID>
13 Feb
</TAF_ID>
<SNAPSHOT/>
<TYPE>
system
</TYPE>
<TIME>
2015-02-17 09:48:13.826
</TIME>
<MACHINE>
qas01
</MACHINE>
<RESULT>
PASS
</RESULT>
<FAILURES/>
</TEST>
5.4 TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE
The TAF configuration file contains all information required by the TAF to checkout, build and execute
tests on specific virtual machine snapshots. Whenever the ETE starts up a new virtual machine snapshot
instance it will copy the TAF and configuration file onto the snapshot along with the snapshot name. The
TAF will then use the snapshot name as an index to lookup the corresponding entry in the configuration
file, and checkout and build the specified versions of software. The configuration entry may also contain
test tags, which the TAF will use to execute the tests required. It may be possible that an entry contains
no test tags in which case the TAF will not execute any tests. This might be required when the tests are
executing on a different virtual machine snapshot but need to start up containers on other machines. The
table below describes the tags used in the TAF configuration file.
Tag Description
<TAF> An instance of the TAF.
<ID> An ID that will help identify output reports.
<INST_DIR> Specify where the DKIST development tree should be checked out.
For example if /opt is specified the atst module will be checked out
to /opt/atst. If <ATST_DIR> is specified, then the ASDT will be
checked out to <INST_DIR>/<ATST_DIR>.
This tag and the <ATST_DIR> tag are also used to set ATST
environment variable for running tests. The ATST environment
QA Software Test Plan
PROC-0015, Revision A Page 35 of 88
variable will be set to <INST_DIR>/<ATST_DIR>. If
<ATST_DIR> is not specified then it will be set to
<INST_DIR>/atst.
<ATST_DIR> Specify a directory name if ‘atst’ should be checked out to a non-
standard directory. The TAF will use the ‘–d’ cvs option to checkout
the ATST Software Development Tree (ASDT). If this tag is not
specified, the ASDT will be checked out to a standard directory
name. (“atst”)
<VM> An instance of the TAF will execute on this virtual machine
snapshot. The virtual machine name is given.
<REPORT> The location to use for storing test report files.
<SOFTWARE> Define which modules and the version to be checked out. If no
version is specified then the module will not be checked out. To
specify the latest updates in the repository set the version to HEAD.
If no SOFTWARE tag is specified then a warning message will be
printed but execution will continue.
The module name should be set as a tag name and the version as a
text. For example, to checkout trunk version of a module ‘qas’, it
should be specified as <qas>HEAD</qas>
There are five special module names: CSF, BASE, QAS, TCS and
MCS
<CSF> The version of CSF software to checkout.
<BASE> The version of BASE software to checkout.
<TCS> The version of TCS software to checkout.
<QAS> The version of the QAS software to check out.
<MCS> The version of the MCS software to checkout.
<module name> Specify a module name as a tag and the version to be checked out as
text. For example, if <mcs>HEAD</mcs> is set in the
SOFTWARE element, the TAF will checkout the trunk version of
the ‘mcs’ module. Also see 5.2.7checkout_directories.
<INSTALL_APP> Configure whether atst should be checked out and built. If the atst
module is already checked out specify 'no'. When 'no' is set the TAF
will ignore <SOFTWARE>, <SITE_CONFIG>,
<CREATE_DEVEL_ACTION> and <MAKE_ACTION>
<SITE_CONFIG> When building atst, the TAF will copy admin/site.config.template to
admin/site.config and replace values of items in site.config.template
as specified in this section. Specify the name of the item as a tag
name and the value as the text. For example there is an item 'baseDir'
in the site.config.template. To set a value for this item do
<baseDir>/opt/atst</baseDir>
All the items that are not set in the template must be specified here or
the build step of the TAF will fail.
If no SITE_CONFIG tag is found then a warning message will be
printed but execution will continue
<CREATE_DEVEL_ACTION> During the build step, TAF will call admin/createDevel after copying
the site.config file. Specify the parameter which should be called
with this call. It will normally be '--make-all' and this is the default
<MAKE_ACTION> During the build step TAF will call 'make' after the
admin/createDevel call. Specify the parameter which should be
QA Software Test Plan
PROC-0015, Revision A Page 36 of 88
passed with this call. It is normally 'build-all' and if nothing is
specified this is used as the default.
<CHECK_POSTGRESQL> As a default the TAF checks whether a postgresql service is running.
The check can be skipped by setting ‘No’ to this tag. This can be
useful when running a test on a machine that does not host the atst
database.
<POSTGRESQL> To run tests or to call init-db the postgresql service must be running.
The TAF will check if the postgresql service is running by calling
'service <service name> status'. Specify the exact name of the
postgresql service here. This may need the version to abe appended
e.g. postgresql-9.3. If nothing is specified then the service name
'postgresql' is used as the default.
<ICESERVICE> The ICE services have to be running or to create the atst database.
The TAF will call startIceServices unless 'no' is specified here
<INIT-DB> The TAF will call ./admin/createDevel --init-db tp create the atst
database after building the atst. if its already been done then this step
can be skipped by specifying 'no' here.
<TESTS> Define the tests (if any) that are to be executed.
<UNIT> Unit tests to be executed.
<SYSTEM> System tests to be executed.
<JAVA> The location of a Java test script file to execute.
<CPP> The location of a C++ test script file to execute.
<PYTHON> The location of a Python test script file to execute.
Below is an example configuration file for the TAF.
<TAF>
<ID>Canary 7-0</ID>
<INST_DIR>/opt</INST_DIR>
<REPORT>/mnt/hgfs/Reports</REPORT>
<SOFTWARE>
<CSF>Canary_7-0</CSF>
<BASE>Canary_7-0</BASE>
<QAS>HEAD</QAS>
</SOFTWARE>
<INSTALL_APP>no</INSTALL_APP>
<SITE_CONFIG>
<baseDir>$ATST</baseDir>
<ATST_RELEASE_FLAG>Canary_7-0</ATST_RELEASE_FLAG>
<ATST_RUN_ENVIRON>/opt/atst/var</ATST_RUN_ENVIRON>
<ATST_JAVA_HOME>/opt/java</ATST_JAVA_HOME>
<ATST_JLIB_DIRS>none</ATST_JLIB_DIRS>
<USE_CPP>yes</USE_CPP>
<ATST_CLIB_DIRS>/usr/pgsql-9.3/lib/:/usr/local/lib/</ATST_CLIB_DIRS>
<ATST_ARCHIVE_DB_HOST>localhost</ATST_ARCHIVE_DB_HOST>
<ATST_ID_DB_HOST>localhost</ATST_ID_DB_HOST>
<ATST_LOG_DB_HOST>localhost</ATST_LOG_DB_HOST>
<ATST_PROPERTY_DB_HOST>localhost</ATST_PROPERTY_DB_HOST>
<ATST_PARAM_SET_DB_HOST>localhost</ATST_PARAM_SET_DB_HOST>
<ATST_SCRIPT_DB_HOST>localhost</ATST_SCRIPT_DB_HOST>
<ATST_HEADER_DB_HOST>localhost</ATST_HEADER_DB_HOST>
<ATST_ICE_VERSION>3.5.1</ATST_ICE_VERSION>
QA Software Test Plan
PROC-0015, Revision A Page 37 of 88
<ATST_ICE_CONNECTION_HOST>localhost</ATST_ICE_CONNECTION_HOST>
<ATST_ICE_EVENTS_HOST>localhost</ATST_ICE_EVENTS_HOST>
<ATST_ALARM_DB_HOST>localhost</ATST_ALARM_DB_HOST>
</SITE_CONFIG>
<CREATE_DEVEL_ACTION>--make-all</CREATE_DEVEL_ACTION>
<MAKE_ACTION>build_all</MAKE_ACTION>
<POSTGRESQL>postgresql-9.3</POSTGRESQL>
<ICESERVICE>no</ICESERVICE>
<INITDB>no</INITDB>
<TESTS>
<UNIT>
<JAVA>atst.qas.tests.csf.REG_CANARY_7_ATCS_627</JAVA>
<JAVA>atst.qas.tests.csf.REG_CANARY_7_ATCS_632</JAVA>
<CPP>atst/qas/tests/csf/REG_CANARY_7_ATCS_627</CPP>
<CPP>atst/qas/tests/csf/REG_CANARY_7_ATCS_636</CPP>
</UNIT>
<SYSTEM>
<PYTHON>$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py</PYTHON>
<PYTHON>$ATST/src/jython/atst/qas/tests/csf/REG_CANARY_7_ATCS_534.py</PYTHON>
</SYSTEM>
</TESTS>
</TAF>
5.4.1 Testing Automation Framework Configuration File Editing Tool
The TAF configuration file can be edited by the configure_taf program. It can create a new TAF
configuration file, edit an existing file, list current settings in the file or check if the file can be used by
TAF. The usage of configure_taf is:
./configure_taf <operation parameter> <options>
configure_taf reads a file $ATST_TEST/src/python/editor/taf_defaults. The file stores default values
which will be used when creating a new TAF configure file. The file should be edited before using the –
new command.
5.4.1.1 ‘—help’ command
Usage: configure_taf –help
Specify –help parameter to see the help message.
./configure_taf –help
---------------------------TAF Config Editor---------------------------
This python program helps creating/editing TAF config files.
Usage: ./configure_taf <command> <options>
Available commands are:
--new Create a new TAF config file.
--edit Updates a TAF config file.
--list List current setting of a TAF config file
--check Check a TAF config file
--siteconfcopy Update a TAF config file's site conf items from an existing
site.conf file
--siteconf Replace the existing site conf items in the specified TAF
config file with the specified items.
QA Software Test Plan
PROC-0015, Revision A Page 38 of 88
If no item is specified, it will delete all site items.
--addsiteconf Add specified site conf items to the specified TAF config file.
--removesiteconf Remove specified site conf items from the specified TAF config
file.
--copy Copy specified TAF config file to create a new TAF config file.
--help Show this message or specify option to show each command's help
message. (i.e. ./configure_test --help add)
Specify –help with an operation parameter will show a detailed help message of the operation.
configure_taf –help siteconfcopy
./configure_taf --help siteconfcopy
Usage: ./configure_taf --siteconfcopy <TAF config file name> --copyfrom=<site.conf
file>
Update site conf items in a TAF config file from an existing site.conf file.
5.4.1.2 ‘—list’ command
Usage: ./configure_taf –list <TAF configuration file path> <options>
To list the items currently set in a TAF configuration file, specify the parameter –list and the file path.
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml
<<<< TAF_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
BASE=HEAD
CSF=HEAD
QAS=HEAD
Unit tests:
JAVA id=1 atst.qas.tests.csf.REG_CANARY_7_ATCS_627
JAVA id=2 atst.qas.tests.csf.REG_CANARY_7_ATCS_632
JAVA id=3 atst.qas.tests.csf.REG_CANARY_7_ATCS_636
CPP id=1 atst/qas/tests/csf/REG_CANARY_7_ATCS_627
CPP id=2 atst/qas/tests/csf/REG_CANARY_7_ATCS_636
CPP id=3 atst/qas/tests/csf/REG_CANARY_7_ATCS_638
System tests:
PYTHON id=1 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
PYTHON id=2 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py
PYTHON id=3 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0106.py
Install application:No
Site config settings:
ATST_ICE_VERSION = 3.5.1
ATST_ID_DB_HOST = localhost
ATST_LOG_DB_HOST = localhost
TST_HEADER_DB_HOST = localhost
USE_CPP = yes
baseDir = $ATST
ATST_BULK_DATA_DB_HOST = localhost
ATST_PARAM_SET_DB_HOST = localhost
ATST_JLIB_DIRS = none
QA Software Test Plan
PROC-0015, Revision A Page 39 of 88
ATST_RELEASE_FLAG = trunk
ATST_ALARM_DB_HOST = localhost
ATST_PROPERTY_DB_HOST = localhost
ATST_RUN_ENVIRON = /opt/atst/var
ATST_ICE_EVENTS_HOST = localhost
TST_SCRIPT_DB_HOST = localhost
ATST_JAVA_HOME = /opt/java8
ATST_ICE_CONNECTION_HOST = localhost
ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST = localhost
createDevel action:--make-all
make action:build_all
Postgresql name:postgresql-9.3
init-db:Yes
Start ICE service:Yes
<<<< TAF_config.xml END >>>>
Items to be listed can be specified. Item options are: id, installdir, reportdir, software, unittests,
systemtests, appinstall, siteconfig, cdaction, maction, postgresql, initdb and ice.
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml id systemtests unittests
<<<< TAF_config.xml START >>>>
ID:New TAF
Unit tests:
JAVA id=1 atst.qas.tests.csf.REG_CANARY_7_ATCS_627
JAVA id=2 atst.qas.tests.csf.REG_CANARY_7_ATCS_632
JAVA id=3 atst.qas.tests.csf.REG_CANARY_7_ATCS_636
CPP id=1 atst/qas/tests/csf/REG_CANARY_7_ATCS_627
CPP id=2 atst/qas/tests/csf/REG_CANARY_7_ATCS_636
CPP id=3 atst/qas/tests/csf/REG_CANARY_7_ATCS_638
System tests:
PYTHON id=1 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
PYTHON id=2 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py
PYTHON id=3 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0106.py
<<<< TAF_config.xml END >>>>
5.4.1.3 ‘—new’ command
Usage: ./configure_taf –new <TAF configuration file path> <options>
To create a new TAF configuration file, call configure_taf with parameter ‘–new’. The TAF configure file
path and –id option is required. Default values for each item are stored in the
$ATST_TEST/src/python/editor/taf_defaults file . When creating a new TAF configure file, the default
values are used for items that are not specified in the options. Specify –nodefaults to not use any default
values.
List of options available to –new are:
--id(Required) The ID for the new TAF config file.
--install_dir Specify where the ATST will be checked out/installed.
(i.e. /opt)
--report_dir Specify where test report should be stored.
(i.e. /opt/Reports).
--software Specify software and its versions to be installed.
(i.e. csf=Canary_7-0,base=Canary_7-0)
--create_devel_action Specify createDevel action. (i.e. --make-all)
--make_action Specify make action. (i.e. build_all)
--install_app Specify whether to checkout and install atst.(yes or no)
--java_unit_test Specify JAVA unit tests, separated by ','.
(i.e. atst.qas.tests.csf.REG_CANARY_7_ATCS_627)
QA Software Test Plan
PROC-0015, Revision A Page 40 of 88
--java_unit_id When specifying JAVA unit tests with --java_unit_test option,
ids for each test can be specified using this option.
--cpp_unit_test Specify CPP unit tests, separated by ','.
(i.e. atst/qas/tests/csf/REG_CANARY_7_ATCS_627)
--cpp_unit_id When specifying CPP unit tests with --cpp_unit_test option,
ids for each test can be specified using this option.
--python_system_test Specify python system tests, separated by ','.
(i.e. '$ATST'/src/jython/atst/qas/tests/csf/SYS_0101.py)
--python_system_id When specifying Python system tests with --python_system_test
option,ids for each test can be specified using this option.
--postgresql Specify postgresql program name. (i.e. posgresql-9.3)
--initdb Specify whether to initialize database. (yes or no)
--iceservice Specify whether to call startIceServices. (yes or no)
--nodefaults Specify this option to not use any default values.
For the example below, the taf_default file is set as follows.
[atst-qas@qas01 bin]$ more $ATST_TEST/src/python/editor/taf_defaults
INST_DIR=/opt
REPORT=/mnt/hgfs/Reports
CREATE_DEVEL_ACTION=--make-all
MAKE_ACTION=build_all
POSTGRESQL=postgresql-9.3
INITDB=Yes
ICESERVICE=Yes
INSTALL_APP=No
#Software defaults should be placed between 'SOFTWARES_START' and 'SOFTWARES_END'
#It should be specified as <Software name>=<Version>
SOFTWARES_START
CSF=HEAD
BASE=HEAD
QAS=HEAD
SOFTWARES_END
#Site config default items should be placed between 'SITE_CONFIG_START' and
'SITE_CONFIG_END'
SITE_CONFIG_START
baseDir=$ATST
ATST_RELEASE_FLAG=trunk
ATST_RUN_ENVIRON=/opt/atst/var
ATST_JAVA_HOME=/opt/java8
ATST_JLIB_DIRS=none
USE_CPP=yes
ATST_CLIB_DIRS=/usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST=localhost
ATST_ID_DB_HOST=localhost
ATST_LOG_DB_HOST=localhost
ATST_PROPERTY_DB_HOST=localhost
ATST_PARAM_SET_DB_HOST=localhost
TST_SCRIPT_DB_HOST=localhost
TST_HEADER_DB_HOST=localhost
ATST_ICE_VERSION=3.5.1
ATST_ICE_CONNECTION_HOST=localhost
ATST_ICE_EVENTS_HOST=localhost
ATST_ALARM_DB_HOST=localhost
ATST_BULK_DATA_DB_HOST=localhost
SITE_CONFIG_END
#System tests should be placed between 'SYSTEM_TESTS_START' and 'SYSTEM_TESTS_END'
#System test should be specified as <TEST path>, <id(optional)>
SYSTEM_TESTS_START
QA Software Test Plan
PROC-0015, Revision A Page 41 of 88
$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py,1
$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py,2
$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0106.py,3
SYSTEM_TESTS_END
#Java Unit tests should be placed between 'JAVA_TESTS_START' and 'JAVA_TESTS_END'
#Java Unit test should be specified as <TEST class path>, <id(optional)>
JAVA_TESTS_START
atst.qas.tests.csf.REG_CANARY_7_ATCS_627,1
atst.qas.tests.csf.REG_CANARY_7_ATCS_632,2
atst.qas.tests.csf.REG_CANARY_7_ATCS_636,3
JAVA_TESTS_END
#Cpp Unit tests should be placed between 'CPP_TESTS_START' and 'CPP_TESTS_END'
#Cpp Unit test should be specified as <TEST class path>, <id(optional)>
CPP_TESTS_START
atst/qas/tests/csf/REG_CANARY_7_ATCS_627,1
atst/qas/tests/csf/REG_CANARY_7_ATCS_636,2
atst/qas/tests/csf/REG_CANARY_7_ATCS_638,3
CPP_TESTS_END
The example below shows how to create a new TAF configuration file using default values. Only the –id
option is specified but other tags are added using default values.
[atst-qas@qas01 bin]$ ./configure_taf --new TAF_config.xml --id="New TAF"
New TAF configure file (TAF_config.xml) created.
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml
<<<< TAF_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
BASE=HEAD
CSF=HEAD
QAS=HEAD
Unit tests:
JAVA id=1 atst.qas.tests.csf.REG_CANARY_7_ATCS_627
JAVA id=2 atst.qas.tests.csf.REG_CANARY_7_ATCS_632
JAVA id=3 atst.qas.tests.csf.REG_CANARY_7_ATCS_636
CPP id=1 atst/qas/tests/csf/REG_CANARY_7_ATCS_627
CPP id=2 atst/qas/tests/csf/REG_CANARY_7_ATCS_636
CPP id=3 atst/qas/tests/csf/REG_CANARY_7_ATCS_638
System tests:
PYTHON id=1 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
PYTHON id=2 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py
PYTHON id=3 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0106.py
Install application:No
Site config settings:
ATST_ICE_VERSION = 3.5.1
ATST_ID_DB_HOST = localhost
ATST_LOG_DB_HOST = localhost
TST_HEADER_DB_HOST = localhost
USE_CPP = yes
baseDir = $ATST
ATST_BULK_DATA_DB_HOST = localhost
ATST_PARAM_SET_DB_HOST = localhost
ATST_JLIB_DIRS = none
ATST_RELEASE_FLAG = trunk
ATST_ALARM_DB_HOST = localhost
ATST_PROPERTY_DB_HOST = localhost
ATST_RUN_ENVIRON = /opt/atst/var
QA Software Test Plan
PROC-0015, Revision A Page 42 of 88
ATST_ICE_EVENTS_HOST = localhost
TST_SCRIPT_DB_HOST = localhost
ATST_JAVA_HOME = /opt/java8
ATST_ICE_CONNECTION_HOST = localhost
ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST = localhost
createDevel action:--make-all
make action:build_all
Postgresql name:postgresql-9.3
init-db:Yes
Start ICE service:Yes
<<<< TAF_config.xml END >>>>
An example of creating a new configure file without using default values.
[atst-qas@qas01 bin]$ ./configure_taf --new TAF_config_2.xml --id="New TAF" --install_
dir="/opt" --report_dir=/mnt/hgfs/Reports --software=CSF=Canary_8,BASE=Canary_8,QAS=
HEAD --nodefaults
New TAF configure file (TAF_config_2.xml) created.
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config_2.xml
<<<< TAF_config_2.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=Canary_8
BASE=Canary_8
QAS=HEAD
<<<< TAF_config_2.xml END >>>>
5.4.1.4 ‘—edit’ command
Usage: ./configure_taf –edit <TAF configure file> <options>
To edit an existing TAF configuration file, call configure_taf with parameter ‘–edit’ and the TAF
configure file’s path followed by options. The options available to –edit option are:
--id The ID of the TAF config file.
--install_dir Specify where the ATST will be checked out/installed.
(i.e. /opt).
--report_dir Specify where test report should be stored.
(i.e. /opt/Reports).
--software Replace the software entry with the specified values.
(i.e. csf=Canary_7-0,base=Canary_7-0)
--add_software Add specified software entry to the existing softwares.
(i.e. csf=Canary_7-0,base=Canary_7-0) If the specified
software already exists, it will be overwitten.
--remove_software Remove the software entry from the existing softwares.
(i.e. csf,base)
--create_devel_action Specify createDevel action. (i.e. --make-all)
--make_action Specify make action. (i.e. build_all)
--install_app Specify whether to checkout and install atst.
(yes or no)
--java_unit_test Replace the JAVA unit tests entry.
(i.e. atst.qas.tests.csf.REG_CANARY_7_ATCS_627)
--add_java_unit_test Add JAVA unit tests entry.
(i.e. atst.qas.tests.csf.REG_CANARY_7_ATCS_627)
--rem_java_unit_test Remove specified JAVA unit tests.
(i.e. atst.qas.tests.csf.REG_CANARY_7_ATCS_627)
--cpp_unit_test Replace the CPP unit tests.
QA Software Test Plan
PROC-0015, Revision A Page 43 of 88
(i.e. atst/qas/tests/csf/REG_CANARY_7_ATCS_627)
--add_cpp_unit_test Add CPP unit tests, separated by ','.
(i.e. atst/qas/tests/csf/REG_CANARY_7_ATCS_627)
--rem_cpp_unit_test Remove specify CPP unit tests, separated by ','.
(i.e. atst/qas/tests/csf/REG_CANARY_7_ATCS_627)
--python_system_test Replace the python system tests, separated by ','.
(i.e. '$ATST'/src/jython/atst/qas/tests/csf/SYS0101.py)
--add_python_system_test Add python system tests, separated by ','.
--rem_python_system_test Remove specified python system tests, separated by ','.
--postgresql Specify postgresql program name. (i.e. posgresql-9.3)
--initdb Specify whether to initialize database. (yes or no)
--iceservice Specify whether to call startIceServices. (yes or no)
This is an example of updating ‘software’ and ‘iceservice’:
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config_2.xml
<<<< TAF_config_2.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=Canary_8
BASE=Canary_8
QAS=HEAD
<<<< TAF_config_2.xml END >>>>
[atst-qas@qas01 bin]$ ./configure_taf --edit TAF_config_2.xml --software=CSF=HEAD,BASE
=HEAD,QAS=HEAD --iceservice=Yes
TAF config file (TAF_config_2.xml) is saved.
[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config_2.xml
<<<< TAF_config_2.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Start ICE service:yes
<<<< TAF_config_2.xml END >>>>
5.4.1.5 ‘—check’ command
Usage: ./configure_taf –check <TAF configuration file path>
Specify the –check parameter to check if a TAF configuration file will work with TAF program. It will
check if any required tags are missing and that the content of the file is well formed.
./configure_taf --check ../../Configs/TAF_config_Canary8_example.xml
TAF config file OK.
./configure_taf --check ../../Configs/TAF_config_2.xml
TAF Config file chek FAILED - Exception occured while reading the TAF configuration
file.
<class 'xml.parsers.expat.ExpatError'>
not well-formed (invalid token): line 27, column 2
QA Software Test Plan
PROC-0015, Revision A Page 44 of 88
./configure_taf --check TAF_config_3.xml
Warning:
ID tag not found in the configuration file
5.4.1.6 Command to edit SITE_CONFIG items
5.4.1.6.1 –siteconfcopy command
Usage: ./configure_taf –siteconfcopy <TAF configuration file path> --copyfrom=<site.config file path>
SITE_CONFIG items can be copied into a TAF configure file from an existing site.config file. Specify –
siteconfcopy parameter, the TAF configuration file path and the site.config file’s path.
./configure_taf --siteconfcopy ../../Configs/taf_config.xml --
copyfrom=../../Configs/site.config
TAF config file (../../Configs/taf_config.xml) is saved.
./configure_taf --list ../../Configs/taf_config.xml
<<<< ../../Configs/taf_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Site config settings:
ATST_ICE_CONNECTION_HOST = localhost
baseDir = $ATST
ATST_PARAM_SET_DB_HOST = localhost
ATST_CPP_CPP_BIN = /opt/rh/devtoolset-2/root/usr/bin/g++
ATST_ALARM_DB_HOST = localhost
ATST_RUN_ENVIRON = /opt/atst/var
ATST_ICE_LIB_SUFFIX = 35
ATST_ID_DB_HOST = localhost
ATST_JAVAC_FLAGS = "-Xlint:all,-path,-serial"
ATST_ICE_JAR =
/usr/share/java/Ice.jar:/usr/share/java/IceStorm.jar:/usr/share/java/IceGrid.jar:/usr/
share/java/Glacier2.jar
ATST_ICE_EVENTS_HOST = localhost
ATST_LOG_DB_HOST = localhost
ATST_ICE_EVENTS_PORT = 12000
ATST_JAVA_DOXYGEN = no
ATST_JAVA_ICE_LIB_PATH = /usr/share/java
ATST_TARGET_ARCH = $(uname -m)
ATST_JAVA_HOME = /opt/java8
ATST_ICE_CONNECTION_PORT = 11000
USE_ICE = yes
ATST_CONCURRENT_CMAKES = 0
ATST_CCINC_DIRS = /usr/local/include/zdb
ATST_BULK_DATA_DB_HOST = localhost
ATST_ICE_VERSION = 3.5.1
ATST_JLIB_DIRS = none
ATST_RELEASE_FLAG = trunk
ATST_JAVA_FLAGS = "-XX:+UseConcMarkSweepGC -Xmx4G"
ATST_PACKAGE_LIST = "base"
ATST_SCRIPT_DB_HOST = localhost
QA Software Test Plan
PROC-0015, Revision A Page 45 of 88
ATST_DB_JAR = /usr/share/java/db-5.3.21.jar
ATST_EXPERIMENT_DB_HOST = none
USE_JAVA = yes
ATST_ARCHIVE_DB_HOST = localhost
ATST_HEADER_DB_HOST = localhost
USE_CPP = yes
ATST_CC_BIN = /opt/rh/devtoolset-2/root/usr/bin/gcc
ATST_JNI_PACKAGES = "none"
ATST_ICE_ICEBOX_PORT = 11998
ATST_ICE_HOME = /usr
ATST_PROPERTY_DB_HOST = localhost
ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/
Start ICE service:yes
<<<< ../../Configs/taf_config.xml END >>>>
5.4.1.6.2 –siteconf command
Usage: ./configure_taf –siteconf <TAF configuration file path> <SITE_CONFIG items>
Specify –siteconf parameter to replace existing SITE_CONFIG items in a TAF configure file.
SITE_CONFIG items should be specified in the format <name>=<value>.
./configure_taf --siteconf ../../Configs/taf_config.xml
ATST_ICE_CONNECTION_HOST=localhost baseDir='$ATST' USE_CPP=no
TAF config file (../../Configs/taf_config.xml) is saved.
./configure_taf --list ../../Configs/taf_config.xml
<<<< ../../Configs/taf_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Site config settings:
USE_CPP = no
baseDir = $ATST
ATST_ICE_CONNECTION_HOST = localhost
Start ICE service:yes
<<<< ../../Configs/taf_config.xml END >>>>
5.4.1.6.3 – addsiteconf command
Usage: ./configure_taf –addsiteconf <TAF configuration file path> <SITE_CONFIG items>
Specify the –addsiteconf parameter to add more SITE_CONFIG items to a TAF configure file. If the
specified SITE_CONFIG item already exists in the TAF configure file, the value of the item will be
updated.
./configure_taf --addsiteconf ../../Configs/taf_config.xml USE_CPP=yes ATST_ICE_CONNEC
TION_PORT=11000 ATST_JAVA_HOME=/opt/java8
TAF config file (../../Configs/taf_config.xml) is saved.
./configure_taf --list ../../Configs/taf_config.xml
QA Software Test Plan
PROC-0015, Revision A Page 46 of 88
<<<< ../../Configs/taf_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Site config settings:
USE_CPP = yes
baseDir = $ATST
ATST_ICE_CONNECTION_HOST = localhost
ATST_ICE_CONNECTION_PORT = 11000
ATST_JAVA_HOME = /opt/java8
Start ICE service:yes
<<<< ../../Configs/taf_config.xml END >>>>
5.4.1.6.4 – removesiteconf command
Usage: ./configure_taf –removesiteconf <TAF configuration file path> <SITE_CONFIG item names>
Specify the –removesiteconf parameter to remove SITE_CONFIG items from a TAF configure file.
./configure_taf --removesiteconf ../../Configs/taf_config.xml USE_CPP ATST_ICE_CONNEC
TION_PORT
TAF config file (../../Configs/taf_config.xml) is saved.
./configure_taf --list ../../Configs/taf_config.xml
<<<< ../../Configs/taf_config.xml START >>>>
ID:New TAF
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Site config settings:
baseDir = $ATST
ATST_ICE_CONNECTION_HOST = localhost
ATST_JAVA_HOME = /opt/java8
Start ICE service:yes
<<<< ../../Configs/taf_config.xml END >>>>
5.4.1.7 –copy command
Usage: ./configure_taf –copy <Existing TAF configuration file path> --newfile=<New TAF configuration
file path> --id=<new ID>
Specify the –copy parameter to create a new TAF configure file by copying an existing TAF configure
file. Specify the new file’s path with the –newfile option and the --id option to set a new ID in the new
file.
./configure_taf --copy ../../Configs/taf_config.xml --newfile=../../Configs/taf_config
_new.xml --id='New TAF 2'
TAF configure file (../../Configs/taf_config_new.xml) created.
QA Software Test Plan
PROC-0015, Revision A Page 47 of 88
./configure_taf --list ../../Configs/taf_config_new.xml
<<<< ../../Configs/taf_config_new.xml START >>>>
ID:New TAF 2
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Site config settings:
baseDir = $ATST
ATST_ICE_CONNECTION_HOST = localhost
ATST_JAVA_HOME = /opt/java8
Start ICE service:yes
<<<< ../../Configs/taf_config_new.xml END >>>>
5.4.1.8 Interactive configuration tool configure_tests
The configure_tests program can be used to edit TAF configuration files interactively. Below is an
example of creating and editing a new TAF configuration file. Pressing Ctrl+D will cancel current
editing process and will go back to a previous menu. When editing, the changes that are made to the TAF
configuration will not be saved to the file until ‘Save’ is selected. ‘*’ will appear next to the file name to
indicate TAF configuration has unsaved changes. This program also makes use of the default file
$ATST_TEST/src/python/editor/taf_defaults
./configure_tests
--------------configure_tests--------------
(Ctrl+D can be used to cancel current editing process)
Create New file or Edit existing file (New/Edit)? >n
Which config file to create? (T)AF or (E)TE)? >t
Creating new TAF config file.
Please specify the new TAF config file path >../../new_taf_config
note:Extension '.xml' is added to the specified file name
Creating new TAF config file.
ID >Created by configure_tests
Install directory >/opt
Report file's directory >/mnt/hgfs/Reports
New file <../../new_taf_config.xml> created
Use Edit menu to add more items.
-------Editing TAF config file < new_taf_config.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>3
<<<< ../../new_taf_config.xml START >>>>
ID:Created by configure_tests
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
<<<< ../../new_taf_config.xml END >>>>
-------Editing TAF config file < new_taf_config.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>ch
TAF config file OK.
QA Software Test Plan
PROC-0015, Revision A Page 48 of 88
-------Editing TAF config file < new_taf_config.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>1
---Edit items <new_taf_config.xml> ------
Available items to edit are
1: id 2: install_dir 3: report_dir 4: software
5: cdaction 6: maction 7: appinstall 8: java_unit_test
9: cpp_unit_test 10: python_system_test
11: postgresql 12: init-db 13: iceservice 14: siteconf
Item to update (Just hit enter to quit editing itmes.) > 4
Current Software:''
Software: Add/update, Delete, Replace, List? >a
Software name > CSF
version > Canary_8
Softwares updated.
Softwares to add > BASE
version > Canary_8
Softwares updated.
Softwares to add > QAS
version > HEAD
Softwares updated.
Softwares to add >
Software: Add/update, Delete, Replace, List? >l
Softwares:
CSF=Canary_8
BASE=Canary_8
QAS=HEAD
Software: Add/update, Delete, Replace, List? >
---Edit items <new_taf_config.xml*> ------
Available items to edit are
1: id 2: install_dir 3: report_dir 4: software
5: cdaction 6: maction 7: appinstall 8: java_unit_test
9: cpp_unit_test 10: python_system_test
11: postgresql 12: init-db 13: iceservice 14: siteconf
Item to update (Just hit enter to quit editing itmes.) >
End editing.
-------Editing TAF config file < new_taf_config.xml* >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>l
<<<< ../../new_taf_config.xml START >>>>
ID:Created by configure_tests
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=Canary_8
BASE=Canary_8
QAS=HEAD
<<<< ../../new_taf_config.xml END >>>>
-------Editing TAF config file < new_taf_config.xml* >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>s
TAF config file (../../new_taf_config.xml) is saved.
-------Editing TAF config file < new_taf_config.xml >-------
QA Software Test Plan
PROC-0015, Revision A Page 49 of 88
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>cf
Create New file or Edit existing file (New/Edit)? >e
Please specify the config file name to edit. >
../../Configs/TAF_config_Canary8_example.xml
-------Editing TAF config file < TAF_config_Canary8_example.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>3
<<<< ../../Configs/TAF_config_Canary8_example.xml START >>>>
ID:Canary 8 Checkout and tests
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=Canary_8
BASE=Canary_8
QAS=HEAD
Unit tests:
JAVA atst.qas.tests.csf.REG_CANARY_8_ATCS_791
JAVA atst.qas.tests.csf.REG_CANARY_8_ATCS_792
CPP atst/qas/tests/csf/REG_CANARY_8_ATCS_791
CPP atst/qas/tests/csf/REG_CANARY_8_ATCS_792
System tests:
PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py
PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0106.py
Install application:yes
Site config settings:
baseDir = $ATST
ATST_RELEASE_FLAG = Canary_8
ATST_RUN_ENVIRON = /opt/atst/var
ATST_JAVA_HOME = /opt/java8
ATST_JLIB_DIRS = none
USE_CPP = yes
ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST = localhost
ATST_ID_DB_HOST = localhost
ATST_LOG_DB_HOST = localhost
ATST_PROPERTY_DB_HOST = localhost
ATST_PARAM_SET_DB_HOST = localhost
ATST_SCRIPT_DB_HOST = localhost
ATST_HEADER_DB_HOST = localhost
ATST_ICE_VERSION = 3.5.1
ATST_ICE_CONNECTION_HOST = localhost
ATST_ICE_EVENTS_HOST = localhost
ATST_ALARM_DB_HOST = localhost
ATST_BULK_DATA_DB_HOST = localhost
createDevel action:--make-all
make action:build_all
Postgresql name:postgresql-9.3
init-db:yes
Start ICE service:yes
<<<< ../../Configs/TAF_config_Canary8_example.xml END >>>>
-------Editing TAF config file < TAF_config_Canary8_example.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>2
TAF config file OK.
QA Software Test Plan
PROC-0015, Revision A Page 50 of 88
-------Editing TAF config file < TAF_config_Canary8_example.xml >-------
1:Edit 2:CHeck 3:List 4:ChangeFile
5:Save 6:Help 7:Quit
>7
QA Software Test Plan
PROC-0015, Revision A Page 51 of 88
6. E2E TEST EXECUTOR
The E2E Test Executor (ETE) is responsible for scheduling automated test runs, setting up and running
the required virtual machines and final processing of the report. The ETE is formed as a set of
scripts/programs that are executed as a cron job and can be easily configured from the command line. A
full explanation of the configuration options is provided in section 6.3. The ETE can be configured to
start up a test run at any pre-defined time on any available virtual machines. When the specified time is
reached the ETE will wake up, start the necessary virtual machines and then copy the TAF scripts and
configuration files to the virtual machines. After the copies have completed the ETE will execute TAF
scripts on the virtual machines and wait for the completion of all tests. Once the tests have been
completed the ETE will shut down the virtual machines and then perform the appropriate notifications of
the test results. The ETE will be responsible for ensuring only one test run is performed at any one time
on one virtual machine, but will not stop multiple virtual machines from being active and running
separate test runs at any one time. From the command line it will be possible for an operator to invoke
the ETE to start a test run immediately.
For situations where real hardware might be used for testing the ETE would not be required, instead
configured instances of the TAF would be manually executed on physical machines. The result would be
a full set of tests run with real hardware and reports stored in the configured location.
6.1 END TO END SIMULATOR
The end-to-end simulator (E2E) is described as the software and hardware system used to integrate and
test all software component, assemblies, and systems that are part of the DKIST control system. The E2E
consists of three servers located in the DKIST Tucson office, and the software QA will form a set of
virtual machine snapshots along with the tools necessary to execute on one of those machines. For
network performance tests it may be necessary to expand and have one extra virtual machine snapshot on
a separate physical machine, separated by a physical network.
The E2E does not contain any hardware required for the subsystems, and so each subsystem must be
executed in a fully simulated state. Any special procedures necessary to execute the subsystems in a
simulated state must be possible through the use of scripts so that the TAF can automate this setting of
subsystems into their simulated state.
6.2 RUNNING THE ETE
The following steps describe how to run the ETE on a machine that has VMWare installed and at least
one prepared Virtual Machine.
1. Checkout the atst_test repository onto the host machine.
export CVSROOT=:pserver:<username>@maunder.tuc.noao.edu:/home/atst/src/qas
cvs login
cvs checkout atst_test
2. Build and install the atst_test source files.
export ATST_TEST=/opt/atst_test
cd $ATST_TEST
./admin/createDevel --make-all
make install_scripts
QA Software Test Plan
PROC-0015, Revision A Page 52 of 88
3. Make sure that there is a shared folder setup between host and the virtual machine that will be
used for executing the tests. The shared folder is used to store the log files from the test
execution, which preserves any log files for review after the tests have completed and the virtual
machine has been shut down.
4. Optionally, login to the virtual machine, and then cvs login to csf, base and qas with the id “atst-
qas”.
5. Create a TAF configuration file on the host machine. The TAF configuration file will be copied
onto the virtual machine by the ETE application. An example of a TAF configuration file is in
$ATST_TEST/src/python/taf/TAF_config_example.xml
Note that test script files should exist in the virtual machine. (Normally they should exist in cvs
so that they will be checked out). Also regarding the SOFTWARE, if no version is specified for a
module, then that module won’t be checked out. For example,
<SOFTWARE>
<CSF>Canary_7</CSF>
<BASE></BASE>
<TCS>HEAD</TCS>
<QAS>HEAD</QAS>
</SOFTWARE>
In this case, cvs checkout won’t be called for ‘base’.
Also, if <INSTALL_APP>no</INSTALL_APP> is specified, then nothing will be checked out
or built. This is useful for running tests on an already installed system ,where there is no
requirement to start from a completely fresh checkout on each execution of the TAF.
There are tools to edit TAF configuration file. See 5.4.1 for more detail.
6. Create an ETE configuration file on the host machine. An example is in
$ATST_TEST/src/python/ete/examples/ETE_config_example.xml
<REPORT_DIR_PATH_ON_VM> Specify the path of the shared folder on the virtual machine.
<REPORT_DIR_PATH_ON_HOST> Specify the path of the shared folder on the host.
There are tools to edit TAF configuration file. See 6.3.2 for more detail.
7. Run the following command to start the ETE.
./ETE --eteconfig="path to the ETE config file" --vmuser=<vm user name> --
vmpassword=<vm user password>
If cvslogin hasn’t been done at step 4, then the --cvsuser and --cvspassword need to be specified.
By default it runs the vm without GUI. Specify --vmwithgui to get a vm gui.
Also by default it shutdowns the virtual machine after the test, which is quite time consuming.
We can set --noshutdown to leave the virtual machine as it is (running) or --suspend to just
suspend it. Please note that ETE will search for a test instance to be executed at the current time.
QA Software Test Plan
PROC-0015, Revision A Page 53 of 88
To ignore the current time and execute a specific test instance the --testtime option should be
specified.
The options for ETE are:
--eteconfig= Set the ETE config file path. (Required)
--vmuser= Set the virtual machine user. (Required)
--vmpassword= Set the virtual machine user's password. (Required)
--cvsuser= Set CVS user for checking out atst software. (Optional)
--cvspassword= Set CVS password for checking out atst software. (Optional)
--emailpgm= Set which email program to use. 'mail'(using 'localhost' SMTP and
current user ID) or 'gmail'(requires --emailuser and
--emailpassword options.). (Optional)
--emailuser= Set Email user for sending final report only if ‘gmail’ is
selected for --emailpgm. (Optional)
--emailpassword= Set the mail user's password only if ‘gmail’ is selected for
--emailpgm. (Optional)
--noshutdown Specify this if the virtual machine shouldn't be shutdown after
the test.(Optional)
--suspend Specify this to 'suspend' the virtual machine rather than
'shutdown' after the test. This is ignored if --noshutdown is
specified. (Optional)
--vmwithgui Specify this to run the VM with GUI.(Optional)
--ignoretime Specify this to run a test instances whose 'time' is specified as
'NOW'. (Optional)
--testtime Specify this to run a test instances of the specified
time. (Optional)
--ignoretime and --testtime options can't be specified together
--help Show help message
Very briefly, how ETE works is:
ETE starts the virtual machine.
ETE copies TAF source code and TAF configuration file into the virtual machine’s home directory.
ETE runs the TAF.sh which has just been copied to the virtual machine.
ETE waits for the signal that TAF has been finished. (There is a lock file in the /tmp folder of the
virtual machine and ETE reads it periodically.)
TAF on the virtual machine reads the configuration file which was copied by ETE.
TAF checks out the modules specified in the config file (unless specified not to).
TAF updates site.config as specified in the config file.(unless specified not to).
TAF builds atst.(unless specified not to).
TAF starts postgresql
TAF starts ICE services (unless specified not to)
TAF does init_db
TAF runs system tests and unit tests.
If there are more VMs to run, the ETE runs the vm and then runs TAF.
When there are no more VMs to run, ETE shuts down all the virtual machines, writes the final report and,
optionally, emails the report.
QA Software Test Plan
PROC-0015, Revision A Page 54 of 88
6.3 CONFIGURATION OF THE ETE
The ETE executes using a set of configuration files. The configuration files are text based xml files that
can be opened by any editor, but a configuration tool is provided to ensure they are correctly formatted.
6.3.1 ETE Configuration File
The ETE configuration file contains all information required for the ETE to be able to start virtual
machine snapshots and to email the results to interested parties. Each instance of ETE execution is
indexed by the time at which the instance should begin execution. The instance contains the names of the
virtual machine snapshots to start (this can be any number of virtual machines) and the email addresses to
send the generated report files to. The number of execution instances is limited only by the requirement
that each instance must begin at a different time. Each instance is indexed by the time and so only one
can exist for each (HH:MM). The ETE will not attempt to start virtual machine snapshots if they are
already running, and will instead report this as an error. The table below describes the tags used in the
ETE configuration file.
Tag Description
<instance> Test run instance. Requires ‘time’ attribute to be set in 24 hour
format HH:MM in UTC to indicate what time the test should
be run. ‘NOW’ can also be specified here to be used with --
ignoretime option.
<EXECUTION> Execution instance. It contains virtual machine instances to run
in the test run instance.
<VM> Virtual Machine instance. A set of information for running
TAF on a virtual machine are specified here. More than one
virtual machine instance can be specified in an execution
instance. Tags which must be specified here are <VM_FILE>,
<TAF_CONFIG>, <REPORT_DIR_PATH_ON_VM>,
<REPORT_DIR_PATH_ON_HOST>,
<SNAPSHOT> and <RUN_ORDER>.
<VM_FILE> Virtual machine file name.
<TAF_CONFIG> Path of the TAF config file to be used on the virtual machine.
<REPORT_DIR_PATH_ON_VM> Path of the shared folder which is set up between the virtual
machine and the host machine to store test reports, on the
virtual machine
<REPORT_DIR_PATH_ON_HOST> Path of the shared folder which is set up between the virtual
machine and the host machine to store test reports, on the host
machine.
<SNAPSHOT> Name of the snapshot of the virtual machine to revert to. It is
possible to leave it empty. In this case, the ETE will not
change the virtual machine state.
<RUN_ORDER> The order of the VM instance to be executed by ETE.
<SYSTEM_TESTS> Optional tag.
Specify the Ids of the system tests which should be executed
by TAF. Multiple Ids can be specified by separating them by
‘,’. ‘-‘ can be used to specify the multiple tests between two
Ids. For example, specifying ‘1-3’ will execute test Ids 1,2, and
3. Specify ‘all’ to run all the system tests.
Steps of each test to be executed can be specified as ‘[1,2,...]’
after each test Ids. For example, to run steps 1,2,3 of a test of
QA Software Test Plan
PROC-0015, Revision A Page 55 of 88
Id 1, then it should be specified as ‘1[1-3]’.
Note: If none of tags <SYSTEM_TESTS>,
<JAVA_UNIT_TESTS> and <CPP_UNIT_TESTS> is present
then TAF will run all the tests specified in the TAF configure
file.
<JAVA_UNIT_TESTS> Optional tag.
Specify the Ids of the Java unit tests which should be executed
by TAF. Multiple Ids can be specified by separating them by
‘,’. ‘-‘ can be used to specify the multiple tests between two
Ids. For example, specifying ‘1-3’ will execute test Ids 1,2, and
3. Specify ‘all’ to run all the system tests.
Note: If none of tags <SYSTEM_TESTS>,
<JAVA_UNIT_TESTS> and <CPP_UNIT_TESTS> is present
then TAF will run all the tests specified in the TAF configure
file.
<CPP_UNIT_TESTS> Optional tag.
Specify the Ids of the CPP unit tests which should be executed
by TAF. Multiple Ids can be specified by separating them by
‘,’. ‘-‘ can be used to specify the multiple tests between two
Ids. For example, specifying ‘1-3’ will execute test Ids 1,2, and
3. Specify ‘all’ to run all the system tests.
Note: If none of tags <SYSTEM_TESTS>,
<JAVA_UNIT_TESTS> and <CPP_UNIT_TESTS> is present
then TAF will run all the tests specified in the TAF configure
file.
<EMAIL_ADDRESS> Email address to send test report to.
<FINAL_REPORT_DIR> The path of the directory where a final report should be stored.
Below is an example configuration file for the ETE. This ETE configure file, at 10:00 UTC ETE will
start up the virtual machine ATST_READY.vmx and run tests specified in the TAF_config_HEAD.xml.
After it completes, it will then startup the virtual machine ATST_READY2.vmx and run the tests
specified in the TAF_config_HEAD2.xml. The ETE will then shutdown both virtual machines and create
the final report in the folder specified in the FINAL_REPORT_DIR tag and email the report to the
address specified in the EMAIL_ADDRESS tag. At 12:40 UTC ETE will start up the virtual machine
ATST_READY.vmx and run the tests specified in the TAF_config_HEAD.xml. When it finishes it will
shut down the virtual machine and create a final report in the folder specified by the
FINAL_REPORT_DIR tag.
<ETE>
<instance time="10:00">
<EXECUTION>
<VM>
<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY.vmx</VM_FILE>
<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_HEAD.xml</TAF_CONFIG>
<REPORT_DIR_PATH_ON_VM>/mnt/hgfs/Reports</REPORT_DIR_PATH_ON_VM>
<REPORT_DIR_PATH_ON_HOST>/drive/ATST_TEST/Reports</REPORT_DIR_PATH_ON_HOST>
<SNAPSHOT>Ready</SNAPSHOT>
<RUN_ORDER>1</RUN_ORDER>
</VM>
<VM>
<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY2.vmx</VM_FILE>
<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_HEAD2.xml</TAF_CONFIG>
<REPORT_DIR_PATH_ON_VM>/mnt/hgfs/Reports</REPORT_DIR_PATH_ON_VM>
<REPORT_DIR_PATH_ON_HOST>/drive/ATST_TEST/Reports</REPORT_DIR_PATH_ON_HOST>
QA Software Test Plan
PROC-0015, Revision A Page 56 of 88
<SNAPSHOT>Ready</SNAPSHOT>
<RUN_ORDER>2</RUN_ORDER>
</VM>
</EXECUTION>
<EMAIL_ADDRESS>Aya <aya@observatorysciences.co.uk> </EMAIL_ADDRESS>
<FINAL_REPORT_DIR>/drive/ATST_TEST/FinalReport</FINAL_REPORT_DIR>
</instance>
<instance time="12:40">
<EXECUTION>
<VM>
<RUN_ORDER>1</RUN_ORDER>
<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY.vmx</VM_FILE>
<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_Canary8.xml</TAF_CONFIG>
<REPORT_DIR_PATH_ON_VM>/mnt/hgfs/Reports</REPORT_DIR_PATH_ON_VM>
<REPORT_DIR_PATH_ON_HOST>/drive/ATST_TEST/Reports</REPORT_DIR_PATH_ON_HOST>
<SNAPSHOT>Ready</SNAPSHOT>
</VM>
</EXECUTION>
<FINAL_REPORT_DIR>/drive/ATST_TEST/FinalReport</FINAL_REPORT_DIR>
</instance>
</ETE>
6.3.2 Configuration Tool
The Configuration tool configure_ete program can be used to edit ETE configuration files to avoid the
necessity of altering the xml files directly. The tool provides abilities to edit, check and list the contents of
an ETE configuration file. The usage of the tool is:
./configure_ete <command parameter> <options>
The configure_ete command uses default values for items that are not specified in the options when
creating a new configure file or editing, whenever possible. The default values are stored in a file
$ATST_TEST/src/python/editor/ete_defaults. The file needs to be updated before running configure_ete.
Available commands are:
--new Create a new empty ETE config file.
--add Add a new test instance.
--copy Copy an ETE config file to create a new ETE config file.
--edit Update items.
--del Delete a test instance.
--addvm Add a new vm instance to a test instance.
--deletevm Delete vm instances from a test instance.
--list List current setting of a ETE config file
--check Check an ETE config file
--changetime Change the execution time of a test instance.
--help Show this message or specify option to show each command's help
message. (i.e. ./configure_test --help add)
6.3.2.1 ‘--list’ command
Usage: ./configure_tests –list <ETE configure file path>
To list all instances of ETE execution use the --list parameter followed by the path of the ETE
configuration file.
QA Software Test Plan
PROC-0015, Revision A Page 57 of 88
[qas@osllx25 bin]$ ./configure_ete --list ../../Configs/ETE_config.xml
Listing ETE config file entries.
Time : 10:00
Final report directory: /drive/ATST_TEST/FinalReport
Email: Aya <aya@observatorysciences.co.uk>
Run order: 1
VM file: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
6.3.2.2 ‘--new’ command
./configure_ete –new <ETE Configure file path>
To create a new empty ETE configuration file, specify the –new parameter with the file path. If the
specified file path doesn’t have any extension, .xml will be added.
./configure_ete --new ../../Configs/ete_conf
note: Extension .xml is added to the file name.
Creating a new empty ETE config file...
New ETE config file (../../Configs/ete_conf.xml) created.
6.3.2.3 ‘--add’ command
Usage: ./configure_ete –new <ETE configuration file path> <options>
To add a new execution instance to the ETE, the --add parameter is used.
The instance is indexed by time. The final report directory and emails can also be specified. Virtual
machine instances can be specified with –vm(n) option. The number ‘n’ represents the run order of the
virtual machine instance. Specify items VM_FILE, TAF_CONFIG, REPORT_DIR_PATH_ON_VM,
REPORT_DIR_PATH_ON_HOST, SNAPSHOT separated by ',' without spaces. Specify --
systemtests_vm(n) to set the SYSTEM_TESTS tag, --javaunittests_vm(n) to set the
JAVA_UNIT_TESTS tag and –cppunittests_vm(n) to set the CPP_UNIT_TESTS tag.
The only required option for the add command is –time. For any of the options that are not specified, the
default values will be used. To avoid using the default values, specify –nodefaults.
The list of available options for the –add command are:
--time(required) The time when the test should run. The format is HH:MM.
--frd The path to the directory where final report will be
stored.
--emails Email addresses. If specified, the final report will be
emailed. Multiple address can be specified separated with
','.
--vm(n) Virtual machine settings. Specify items <VM_FILE>,
<TAF_CONFIG>,<REPORT_DIR_PATH_ON_VM>,
<REPORT_DIR_PATH_ON_HOST>, <SNAPSHOT> separated by ','
without spaces. Specify--vm with number of order. For
example, if adding <VM> with order number of 2, the option
name is --vm2.
QA Software Test Plan
PROC-0015, Revision A Page 58 of 88
--nodefaults Specify this option to not use any default values. If this
option is not specified, default values will be used for
items that are not specified.
--systemtests_vm(n) Specify the IDs of system tests which should be run by TAF.
The ‘n’ represents the order of the VM instance.
--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by
TAF. The ‘n’ represents the order of the VM instance.
--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by
TAF. The ‘n’ represents the order of the VM instance.
For the example below, the ete_default file is set as follows.
VM_FILE=/drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
TAF_CONFIG=/drive/ATST_TEST/Configs/TAF_config_HEAD.xml
REPORT_DIR_PATH_ON_VM=/mnt/hgfs/Reports
REPORT_DIR_PATH_ON_HOST=/drive/ATST_TEST/Reports
FINAL_REPORT_DIR=/drive/ATST_TEST/FinalReport
EMAIL_ADDRESS=aya@observatorysciences.co.uk
SNAPSHOT=Ready
SYSTEM_TESTS=all
JAVA_UNIT_TESTS=1-5
CPP_UNIT_TESTS=all
This example adds a new test instance at 12:00 by specifying only the –time option. It demonstrates that
VM instance with order number 1 is added using default values.
[aya@osllx21 bin]$ ./configure_ete --add ETE.xml --time=12:00
Adding a new test instance...
VM order 1 with default values is added.
Default email addresses are added.
Default final report directory is used.
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 12:00
Final report directory: /drive/ATST_TEST/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
When –nodefaults option is specified, no items other than those specified in the options will be added.
[aya@osllx21 bin]$ ./configure_ete --add ETE.xml --time=14:00 --frd=$ATST_TEST/
FinalReport --emails=aya@observatorysciences.co.uk --nodefaults
Adding a new test instance...
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 12:00
QA Software Test Plan
PROC-0015, Revision A Page 59 of 88
Final report directory: /drive/ATST_TEST/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
---------------end-------------------
6.3.2.4 ‘--del’ command
Usage: configure_ete --del < ETE configure file path > --time <HH:MM>
To remove a previously defined test instance from the ETE, use the --del parameter. The instance to
remove needs to be indexed by time.
[aya@osllx21 bin]$ ./configure_ete --del ETE.xml --time=12:00
Deleting a test instance...
Test instance (12:00) removed.
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
---------------end-------------------
6.3.2.5 ‘--addvm’ command
Usage: ./configure_ete --addvm <ETE configure file path> --time=<time> --order=<VM order>
<options>...
To add a new VM instance to a test instance, the --addvm parameter is used.
--time option is required to specify which test instance the new
VM will be added to. The order of the virtual machine is
specified by the –order option. Other items of the VM
instance can be specified using the options listed below.
Similar to the –add command, the default values will be
used unless –nodefaults option is specified.
Available options for '-addvm' are:
--time(required) The time when the test should run. The format is HH:MM.
--order(required) The order of the new VM instance.
--vmfile The VM file path of the new VM instance.
--tafconfig The TAF configure file path of the new VM instance.
QA Software Test Plan
PROC-0015, Revision A Page 60 of 88
--report_dir_vm The directory for the report on the virtual machine.
--report_dir_host The directory for the report on the host machine.
--snapshot The snapshot name.
--systemtests_vm(n) Specify the IDs of system tests which should be run by TAF.
The ‘n’ represents the order of the VM instance.
--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by
TAF. The ‘n’ represents the order of the VM instance.
--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by
TAF. The ‘n’ represents the order of the VM instance.
--nodefaults Specify this option to not use any default values. If this
option is not specified, default values will be used for
items that are not specified.
This example adds a new VM instance to the test instance at 14:00 by specifying only the –time and –
order option. It shows that a VM instance is added to the order number 1 with default values, then adds a
VM instance with order number 2 using a different TAF configure file.
[aya@osllx21 bin]$ ./configure_ete --addvm ETE.xml --time=14:00 --order=1
Adding/replacing <VM> to a test instance...
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
[aya@osllx21 bin]$ ./configure_ete --addvm ETE.xml --time=14:00 --order=2 --
tafconfig=/drive/ATST_TEST/Configs/TAF_config_Canary_8.xml
Adding/replacing <VM> to a test instance...
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
Run order: 2
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
QA Software Test Plan
PROC-0015, Revision A Page 61 of 88
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_Canary_8.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
6.3.2.6 ‘--delvm’ command
Usage: ./configure_ete --deletevm < ETE configure file path > --time=<time> --orders=<run orders>
To remove a VM instance from a test instance, use the --deletevm parameter. The instance to remove
needs to be specified by test execution time and the order number of the VM instance.
[aya@osllx21 bin]$ ./configure_ete --deletevm ETE.xml --time=14:00 --order=2
VM at the run order 2 deleted.
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: all
Java unit tests: 1-5
CPP unit tests: all
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
6.3.2.7 ‘--list’ command
Usage: ./configure_ete --list < ETE configure file path >
To list all test instances use the --list parameter.
./configure_ete --list /home/qas/ATST_TEST/Configs/ete_config.xml
Listing ETE config file entries.
Time : 12:00
Final report directory: /home/qas/ATST_TEST/atst_test/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/Centos66/Centos66.vmx
Snapshot name: Canary_8_Ready
TAF config file: /home/qas/ATST_TEST/Configs/taf_config.xml
Report directory path on the virtual machine: /mnt/hgfs/Reports
QA Software Test Plan
PROC-0015, Revision A Page 62 of 88
Report directory on the host machine: /home/qas/ATST_TEST/atst_test/../Reports
Run order: 2
VM file: /drive/VMWareWorkstation/Centos66/Centos66_2.vmx
Snapshot name: Canary_8_Readyist
TAF config file: /home/qas/ATST_TEST/Configs/taf_config.xml
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /home/qas/ATST_TEST/atst_test/../Reports
Run order: 3
VM file: /drive/VMWareWorkstation/Centos66/Centos66_3.vmx
Snapshot name: Canary_8_Ready
TAF config file: /home/qas/ATST_TEST/Configs/taf_config.xml
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /home/qas/ATST_TEST/atst_test/../Reports
6.3.2.8 ‘--edit’ command
Usage: ./configure_ete --edit < ETE configure file path > <options>
To edit items in a test instance, use the –edit parameter.
The test instance must be specified by the execution time. Items that need to be updated should be
specified with the options listed below.
--emails Email addresses separated by ','. If '' is set then email
tags will be deleted.
--frd The path to the directory where final report will be
stored.
--vm(n) Virtual machine settings. Specify items <VM_FILE>,
<TAF_CONFIG>, <REPORT_DIR_PATH_ON_VM>,
<REPORT_DIR_PATH_ON_HOST>, <SNAPSHOT> separated by ','
without spaces. The number ‘n’ represents the order of the
VM instance.
--systemtests_vm(n) Specify the IDs of system tests which should be run by TAF.
--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by
TAF.
--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by
TAF.
In the example below, the emails of a test instance at 14:00 and System tests of the VM instance order 1
is updated.
[aya@osllx21 bin]$ ./configure_ete --edit ETE.xml --time=14:00 --systemtests_vm1=10- -
-emails=aya2@observatorysciences.co.uk
Editing the test instance (14:00)...
ETE config file (ETE.xml) is saved.
-----ETE configure file - ETE.xml -----
Time : 14:00
Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport
Email: aya2@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: 10-
Java unit tests: 1-5
CPP unit tests: all
QA Software Test Plan
PROC-0015, Revision A Page 63 of 88
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
---------------end-------------------
6.3.2.9 ‘--check’ command
Usage:./configure_ete –check < ETE configure file path >
Use the –check parameter to check an ETE configure file to see if all required items are set and can be
used by ETE. It checks whether the file is in the correct format and files and directories specified in the
ETE config file exist.
./configure_ete --check /home/qas/ATST_TEST/Configs/ete_config.xml
Checking ETE config file
Error:
instance (time:12:00) VM3 REPORT_DIR_PATH_ON_HOST tag is missing
Warning:
instance (time:12:00) VM1 VM_FILE (/drive/VMWareWorkstation/Centos66/Centos66.vmx)
doesn't exist.
instance (time:12:00) VM1 TAF_CONFIG (/home/qas/ATST_TEST/Configs/taf_config.xml)
doesn't exist.
instance (time:12:00) VM1 REPORT_DIR_PATH_ON_HOST
(/home/qas/ATST_TEST/atst_test/../Reports) doesn't exist.
6.3.2.10 ‘--copy’ command
Usage: ./configure_ete –copy <file path of the ETE configure file to copy> <new ETE configure file
path>
To copy and create a new ETE configure file, specify the –copy parameter followed by the path of the
ETE configure file to copy from and the new ETE configure file path.
[aya@osllx21 bin]$ ./configure_ete --copy ETE.xml ETE_2.xml
Creating a new empty ETE config file...
New ETE config file (ETE_2.xml) created.
6.3.2.11 ‘--help’ command
Usage: ./configure_ete --help
Specify –help to show the help message.
------------------------------ETE Config Editor------------------------------
This python program helps creating/editing ETE config file.
Usage: ./configure_ete <command> <options>
Available commands are:
--new Create a new empty ETE config file.
--add Add a new test instance.
--copy Copy an ETE config file to create a new ETE config file.
--edit Update items.
--del Delete a test instance.
--addvm Add a new vm instance to a test instance.
--deletevm Delete vm instances from a test instance.
--list List current setting of a ETE config file
QA Software Test Plan
PROC-0015, Revision A Page 64 of 88
--check Check an ETE config file
--changetime Change the execution time of a test instance.
--help Show this message or specify option to show each command's help
message. (i.e. ./configure_test --help add)
Specify –help with a command name to show the help message of that command.
./configure_ete –help <command name>
[aya@osllx21 bin]$ ./configure_ete --help add
Usage: ./configure_ete --add <file name> --time=<time> <options>
Create a new test instance in the specified ETE config file.
Available options are :
--time(required) The time when the test should run. The format is HH:MM.
--frd The path to the directory where final report will be stored.
--emails Email addresses. If specified, final report will be sent.
Multiple address can be specified separated with ','.
--vm(n) Virtual machine settings. Specify items <VM_FILE>,<TAF_CONFIG>
<REPORT_DIR_PATH_ON_VM>,<REPORT_DIR_PATH_ON_HOST>,<SNAPSHOT>
separated by ',' without spaces.
Specify--vm with number of order. For example, if adding <VM>
with order number of 2, option name is --vm2.
--nodefaults Specify this option to not use any default values.
If this option is not specified, default values will be used
for items that are not specified.
--systemtests_vm(n) Specify the IDs of system tests which should be run by TAF.
Specify 'all' to run all tests.
--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by TAF.
Specify 'all' to run all tests.
--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by TAF.
Specify 'all' to run all tests.
6.3.2.12 Interactive configure tool
The configure_tests program can be used to edit ETE configuration files interactively. Below is an
example of creating a new ETE configuration file then editing and checking it. Pressing Ctrl+D will
cancel the current editing process and go back to a previous menu. When editing, the changes that are
made to the ETE configuration will not be saved to the file until ‘Save’ is selected. ‘*’ will appear next to
the file name to indicate the configuration has unsaved changes. This program also makes use of the
default file $ATST_TEST/src/python/editor/ete_defaults.
[atst-qas@osllx25 bin]$ ./configure_tests
--------------configure_tests--------------
(Ctrl+D can be used to cancel current editing process)
Create New file or Edit existing file (New/Edit)? >n
Which config file to create? (T)AF or (E)TE)? >e
Creating new ETE config file.
Please specify the new ETE config file path >ete_config
note:Extension '.xml' is added to the specified file name
-------Editing ETE config file <ete_config.xml>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>1
-----Adding new test instance-----
Time(UTC)? >12:00
VM1 VM file name? (default:/drive/VMWareMachines/ATST_READY/ATST_READY.vmx) >
VM1 TAF config file name? (default:/drive/ATST_TEST/Configs/TAF_config_HEAD.xml) >
QA Software Test Plan
PROC-0015, Revision A Page 65 of 88
VM1 Report folder on VM? (default:/mnt/hgfs/Reports) >
VM1 Report folder on host? (default:/drive/ATST_TEST/Reports) >
VM1 Snapshot name? (default:Ready Type 'None' to have no Snapshot set.) >
Specify SYSTEM_TESTS? (yes/No)>y
SYSTEM_TESTS for VM1?(default:all) >1-3
Specify JAVA_UNIT_TESTS? (yes/No)>y
JAVA_UNIT_TESTS for VM1?(default:all) >
Specify CPP_UNIT_TESTS? (yes/No)>y
CPP_UNIT_TESTS for VM1?(default:all) >a
Emails? (default:aya@observatorysciences.co.uk Type 'None' to have no email addresses
set.) >
Final report directory? (default:/drive/ATST_TEST/FinalReport) >
New test instance added.
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>5
Time : 12:00
Final report directory: /drive/ATST_TEST/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
Snapshot name: Ready
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: 1-3
Java unit tests: all
CPP unit tests: a
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>4
Which test instance to edit? (Time 12:00)
(Just hit enter to quit editing VM of the test instance) > 12:00
-----Editing a test instance (12:00)-----
1:Emails 2:Frd(Final report directory) 3:Vm(Edit a VM instance)
4:AddVm(Add VM) 5:DeleteVm(Delete a VM)
6:List(Show current setting of the teset instance)
Edit instance (12:00): What item to edit? >VM
------Editing VM1------
1:VMfile 2:Tafconfig 3:RepoVm 4:RepoHost
5:Snapshot 6:RunOrder 7:List 8:ListTaf
VM1: Which item to edit? >snapshot
Current Snapshot name: Ready
New snapshot name >Ready2
------Editing VM1------
1:VMfile 2:Tafconfig 3:RepoVm 4:RepoHost
5:Snapshot 6:RunOrder 7:List 8:ListTaf
VM1: Which item to edit? >
-----Editing a test instance (12:00)-----
1:Emails 2:Frd(Final report directory) 3:Vm(Edit a VM instance)
4:AddVm(Add VM) 5:DeleteVm(Delete a VM)
6:List(Show current setting of the teset instance)
QA Software Test Plan
PROC-0015, Revision A Page 66 of 88
Edit instance (12:00): What item to edit? >
Which test instance to edit? (Time 12:00)
(Just hit enter to quit editing VM of the test instance) >
Time not specified.
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>5
Time : 12:00
Final report directory: /drive/ATST_TEST/FinalReport
Email: aya@observatorysciences.co.uk
Run order: 1
VM file: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
Snapshot name: Ready2
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
System tests: 1-3
Java unit tests: all
CPP unit tests: a
Report directory path on the virtual machine: /mnt/hgfs/Reports
Report directory on the host machine: /drive/ATST_TEST/Reports
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>6
ETE config file OK.
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>listtaf
<<<< /drive/ATST_TEST/Configs/TAF_config_HEAD.xml START >>>>
ID:HEAD Checkout, Some tests.
Install directory:/opt
Report file directory:/mnt/hgfs/Reports
Softwares:
CSF=HEAD
BASE=HEAD
QAS=HEAD
Unit tests:
JAVA atst.qas.tests.csf.REG_CANARY_7_ATCS_627
JAVA atst.qas.tests.csf.REG_CANARY_7_ATCS_632
JAVA atst.qas.tests.csf.REG_CANARY_7_ATCS_636
CPP atst/qas/tests/csf/REG_CANARY_7_ATCS_627
CPP atst/qas/tests/csf/REG_CANARY_8_ATCS_791
CPP atst/qas/tests/csf/REG_CANARY_8_ATCS_792
System tests:
PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py
PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0103.py
Install application:yes
Site config settings:
baseDir = $ATST
ATST_RELEASE_FLAG = trunk
ATST_RUN_ENVIRON = /opt/atst/var
ATST_JAVA_HOME = /opt/java8
ATST_JLIB_DIRS = none
QA Software Test Plan
PROC-0015, Revision A Page 67 of 88
USE_CPP = yes
ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST = localhost
ATST_ID_DB_HOST = localhost
ATST_LOG_DB_HOST = localhost
ATST_PROPERTY_DB_HOST = localhost
ATST_PARAM_SET_DB_HOST = localhost
ATST_SCRIPT_DB_HOST = localhost
ATST_HEADER_DB_HOST = localhost
ATST_ICE_VERSION = 3.5.1
ATST_ICE_CONNECTION_HOST = localhost
ATST_ICE_EVENTS_HOST = localhost
ATST_ALARM_DB_HOST = localhost
ATST_BULK_DATA_DB_HOST = localhost
createDevel action:--make-all
make action:build_all
Postgresql name:postgresql-9.3
init-db:yes
Start ICE service:yes
<<<< /drive/ATST_TEST/Configs/TAF_config_HEAD.xml END >>>>
-------Editing ETE config file <ete_config.xml*>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>save
ETE config file (ete_config.xml) is saved.
-------Editing ETE config file <ete_config.xml>-------
1:Add 2:Del 3:ChangeTime 4:Edit
5:List 6:Check 7:ListTaf 8:ChangeFile
9:Save 10:Copy 11:Quit 12:Help
>quit
[atst-qas@osllx25 bin]$
6.4 TEST RESULTS AND REPORT GENERATION
Section 5.2.5 describes the report format that is produced for each test executed by the TAF. Once all
tests have been completed the reports are collated and then parsed to produce the final report. The final
report provides the details of virtual machine and TAF configure files used and the execution time. It
then shows a short summary of results for each of the types of test (unit and system). The summary
shows how many tests were executed, how many passed and how many failed. For each test failure
details of the failure are appended to the end of the report to aid in tracing the problem that occurred. An
example report is shown below.
Test Execution: Fri Jan 23 10:10:01 2015
Virtual machine: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx
TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml
Results Summary:
Run Pass Fail
Unit 32 30 2
System 97 97 0
Details of failures:
ID = REG_CANARY_8_ATCS_791.h
CVS ID = $Id: REG_CANARY_8_ATCS_791.h,v 1.1 2014/10/01 09:03:09 ChrisMayer Exp $
VM name = "/drive/VMWareMachines/ATST_READY/ATST_READY.vmx"
Snapshot = "Ready"
QA Software Test Plan
PROC-0015, Revision A Page 68 of 88
TAF ID = "HEAD
Type = unit
Time = 2015/01/23:11:59:14.361 GMT
Pass = False
Machine = qas01
Failures =
Locations =
REG_CANARY_8_ATCS_791.h[140]
Reasons =
Error: Expected (nanoStringOut[i]+gmt == d1->toNanoString()), found
("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")
Versions =
[CSF=HEAD]
[Base=HEAD]
ID = REG_CANARY_8_ATCS_792.h
CVS ID = $Id: REG_CANARY_8_ATCS_792.h,v 1.1 2014/10/01 09:03:18 ChrisMayer Exp $
VM name = "/drive/VMWareMachines/ATST_READY/ATST_READY.vmx"
Snapshot = "Ready"
TAF ID = "HEAD
Type = unit
Time = 2015/01/23:11:59:19.472 GMT
Pass = False
Machine = qas01
Failures =
Locations =
REG_CANARY_8_ATCS_792.h[92]
Reasons =
Error: Expected (nanoStringOut[i] == d1->toNanoString()), found
("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")
Versions =
[CSF=HEAD]
[Base=HEAD]
6.5 CONFIGURING A CRON JOB FOR ETE
You can set up a cron job to run the ETE every day at the time specified in an ETE configure file. The
example below shows a typical cron job setting.
*/30 * * * * export ATST_TEST=/drive/ATST_TEST/atst_test;filename=`eval date -u
+\%Y\%m\%d_\%H\%M\%S`"_ETE.log";/drive/ATST_TEST/atst_test/bin/ETE --
eteconfig=/drive/ATST_TEST/Configs/ETE_config.xml --vmuser=qas --vmpassword=password -
-emailpgm=mail > $ATST_TEST/etelog/$filename
With this cron job setting, the ETE will be called every 30 minutes. If it finds a test instance to execute
for the current time, then it will execute the test. In this cron job, the output will be stored in a file
$ATST_TEST/etelog/<file name>. The file name is constructed as YYYYmmdd_HHMMSS_ETE.log.
QA Software Test Plan
PROC-0015, Revision A Page 69 of 88
7. SETUP OF VMWARE VIRTUAL MACHINE
To ensure a repeatable set of tests are executed each time the testing procedure is carried out virtual
machines is used that provide the same starting point for each test run. A set of virtual machines should
be available, each with a specific setup that can be utilized by the ETE. The virtual machines should be
setup and snapshots taken so that after each test run the virtual machine is returned to a "clean" state, with
no properties or temporary files left over. Two copies of each virtual machine should be available to
allow for networked tests to take place (not network performance tests as these must be executed across
real networks).
7.1.1 Virtual Machine Snapshot Setup
Virtual machines should be created in pairs, one pair for each testing environment. A testing environment
is defined by the operating system running on it (and the versions of some packages), the postgres
database version running, the version of ICE, the version of Java and the version of the GCC compiler. If
any of these environmental setups are changed then another pair of virtual machines should be created for
testing purposes. By maintaining these sets of virtual machines it is easier to verify conditions that might
affect the failure of tests, and this method eliminates the possibility of tests being affected inadvertently
by upgrading operating system packages, or installing other software that appears to be unrelated but
actually isn't. The virtual machines and snapshots should follow a simple naming convention to easily
identify what setup is present on each instance.
7.1.2 Shared Folder
As the tests are executed the only information that should be retained are the test data files and reports. A
shared folder should be created in a virtual machine to store files and reports. Once the tests have all been
executed and the virtual machine has been shut down all of the test reports and data files are available to
the ETE for compilation of the results.
Each execution of the tests will result in a new sub-directory within the main shared folder so that no
previous sets of test reports and data files can be overwritten. Currently it is assumed that manual
archiving of the test data may be required over time but that is outside the scope of this document.
7.1.3 Remote Control of Virtual Machines
For the ETE to automate execution of tests it is necessary to be able to control the virtual machines
remotely. VMWare Workstation provides a remote control tool called ‘vmrun’. This provides the
following operations as command line tools, available for scripting
Power on virtual machines.
Power off virtual machines.
Copy files from host to guest.
Delete files from host to guest.
Rename files from host to guest.
Run programs and scripts in guest.
Using the ‘vmrun’ it is possible to run the required operations on a virtual machine from a host machine’s
terminal. This means the guest machine doesn't need to have any special software or scripts to execute
the tests, all execution is controlled externally.
QA Software Test Plan
PROC-0015, Revision A Page 70 of 88
8. UNIT TESTING
For the purposes of simplifying the DKIST testing procedure it has been decided that instead of
segregating unit tests and assembly tests they shall all be considered as unit tests. A description of the
definition of unit and assembly testing is presented below, after which all tests that fall under these
categories will be considered as unit tests.
Unit testing is a method by which individual units of source code, sets of one or more computer program
modules together with associated control data, usage procedures, and operating procedures, are tested to
determine if they are fit for use. Intuitively, one can view a unit as the smallest testable part of an
application, the level of a class in C++ and Java. Unit tests are usually created by programmers during
the development process and would be considered white box testing. Ideally, each test case is
independent from the others: substitutes like method stubs, mock objects, fakes and test harnesses can be
used to assist testing a module in isolation.
Assembly testing provides the next step up from unit testing. It demonstrates that modules (classes)
interact in a stable and proper manner as defined by the functional specifications. Assembly testing takes
as its input modules that have been unit tested, groups them in larger aggregates, applies the defined
assembly tests defined to those aggregates, and delivers as its output the integrated system ready for
system testing.
For the DKIST software unit testing we require separate testing suites for Java and C++. The selected test
frameworks that will be used for the unit tests are JUnit for the Java codebase and CxxTest for the C++
codebase. Both of these frameworks are open source and actively maintained, and CxxTest has
previously been used to perform some unit tests on the CSF C++ codebase. Neither of these frameworks
are intrusive to the code which allows the software tests to be developed alongside the current DKIST
CSF and Base stable versions without disruption.
Unit tests will only be carried out of CSF and Base code, there are no requirements to perform unit tests
on other systems or subsystems. However, due to the size of the CSF and Base and the effort constraints
it is currently not realistic to expect unit tests to be written for all classes. Therefore a selection of classes
that are considered critical have been selected for unit testing and tests will be written for both languages
(Java and C++).
8.1 WRITING A UNIT TEST
Wherever possible when writing a unit test the test must be added for both Java and C++. Only in the
case of testing differences in the two languages should separate tests be written. To write a new unit test
the following steps should be followed.
8.1.1 Writing a Java Unit Test
1) Identify the name of the new unit test. There is no restriction placed on the naming of tests but if the
name includes a short description of what the test covers it is easier to manage large sets of tests.
2) Create a new Java class under [TBD] the name of which is the same as decided in step 1.
3) Create a test method within the class file and annotate the test method with @org.junit.Test to notify
the JUnit test framework that this is a test method.
QA Software Test Plan
PROC-0015, Revision A Page 71 of 88
4) Write the test. JUnit provides many useful static methods in the Assert class to test for certain
conditions, which throw AssertionExceptions if the comparison fails. The table below lists some of the
Assert static methods available.
Statement Description
fail(String) Let the method fail. Might be used to check that a
certain part of the code is not reached. Or to have a
failing test before the test code is implemented.
assertTrue([message], boolean condition) Checks that the boolean condition is true.
assertsEquals([String message], expected,
actual)
Tests that two values are the same. Note: for arrays the
reference is checked not the content of the arrays.
assertsEquals([String message], expected,
actual, tolerance)
Test that float or double values match. The tolerance is
the number of decimals which must be the same.
assertNull([message], object) Checks that the object is null.
assertNotNull([message], object) Checks that the object is not null.
assertSame([String], expected, actual) Checks that both variables refer to the same object.
assertNotSame([String], expected, actual) Checks that both variables refer to different objects.
The ATSTUnitTestRunner (described in section 4.1) will form the correct report format from the test if it
fails.
5) Add the test to the makefile and then follow the instructions in section 8.4 to add the test to a test set.
8.1.2 Writing a C++ Unit Test
1) Identify the name of the new unit test. There is no restriction placed on the naming of tests but if the
name includes a short description of what the test covers it is easier to manage large sets of tests. If
testing the same functionality as that of a Java unit test then keep the name the same.
2) Create a new C++ header file under [TBD] the name of which is the same as decided in step 1.
3) C++ unit tests are not executed in the same way as JUnit tests, the C++ unit tests must be compiled into
executable files. First the header files are parsed and then the generated code is compiled. Open the
header file.
4) Create a test class which extends the class CxxTest::TestSuite. Add test methods within the class file.
5) Write the test. CxxTests provides several useful macros for assertions to test for conditions. The table
below lists the assertion macros provided by CxxTests.
Macro Description
QA Software Test Plan
PROC-0015, Revision A Page 72 of 88
TS_FAIL(message) Fail unconditionally.
TS_ASSERT(expr) Verify (expr) is true.
TS_ASSERT_EQUALS(x, y) Verify (x==y).
TS_ASSERT_SAME_DATA(x, y, size) Verify two buffers are equal.
TS_ASSERT_DELTA(x, y, d) Verify (x==y) up to d.
TS_ASSERT_DIFFERS(x, y) Verify !(x==y).
TS_ASSERT_LESS_THAN(x, y) Verify (x<y).
TS_ASSERT_LESS_THAN_EQUALS(x, y) Verify (x<=y).
TS_ASSERT_PREDICATE(R, x) Verify P(x).
TS_ASSERT_RELATION(R, x, y) Verify x R y.
TS_ASSERT_THROWS(expr, type) Verify that (expr) throws a specific type of exception.
TS_ASSERT_THROWS_EQUALS(expr,
arg, x, y)
Verify type and value of what (expr) throws.
TS_ASSERT_THROWS_ASSERT(expr,
arg, assertion)
Verify type and value of what (expr) throws.
TS_ASSERT_THROWS_ANYTHING(expr) Verify that (expr) throws an exception.
TS_ASSERT_THROWS_NOTHING(expr) Verify that (expr) doesn't throw anything.
TS_WARN(message) Print message as a warning.
TS_TRACE(message) Print message as an informational message.
6) Add the test to the makefile. When building the build system will produce an executable for the test
that can be run.
7) Follow the instructions in section 8.4 to add the test to a test set.
8.2 EXAMPLE JAVA UNIT TEST
The following example test is taken from the original CSF Java unit tests and is out of date and may not
work. This example should be replaced once we have written some updated test classes.
package atst.cs.test;
import java.util.*;
import org.junit.*;
QA Software Test Plan
PROC-0015, Revision A Page 73 of 88
import atst.cs.interfaces.*;
import atst.cs.services.*;
import atst.cs.data.*;
public class TestData {
@Test
public void attribute() {
Boolean[] bArr = { new Boolean(true), new Boolean(false), new
Boolean(true) };
Double[] dArr = { new Double(Double.MIN_VALUE), new Double(0.0d),
new Double(Double.MAX_VALUE) };
Integer[] iArr = { new Integer(Integer.MIN_VALUE), new
Integer(0),
new Integer(Integer.MAX_VALUE) };
String[] sArr = { "string 1", "string 2", "string 3" };
float[] f_Arr = { Float.MIN_VALUE, 0.0f, Float.MAX_VALUE };
int[] i_Arr = { Integer.MIN_VALUE, 0, Integer.MAX_VALUE };
long[] l_Arr = { Long.MIN_VALUE, 0L, Long.MAX_VALUE };
IAttribute a1, a2, a3, a4, a5, a6, a7, a8, a9, aA, aB, aC, aD;
a1 = new Attribute("valueless");
a2 = new Attribute("BoolArray", bArr);
a3 = new Attribute("DoubleArray", dArr);
a4 = new Attribute("IntArray", iArr);
a5 = new Attribute("string", "a string");
a6 = new Attribute("sArray", sArr);
a7 = new Attribute("bool", true);
a8 = new Attribute("double", Double.MAX_VALUE);
a9 = new Attribute("float", Float.MAX_VALUE);
aA = new Attribute("floatArray", f_Arr);
aB = new Attribute("intArray", i_Arr);
aC = new Attribute("long", Long.MAX_VALUE);
aD = new Attribute("longArray", l_Arr);
int i;
// NOTE: Name value returned is NOT qualified. Spec implied
otherwise
Assert.assertTrue("valueless".equals(a1.getName()));
Assert.assertTrue("BoolArray".equals(a2.getName()));
Assert.assertArrayEquals(bArr, a2.getBooleanArray());
Assert.assertTrue("DoubleArray".equals(a3.getName()));
Assert.assertArrayEquals(dArr, a3.getDoubleArray());
Assert.assertTrue("IntArray".equals(a4.getName()));
Assert.assertArrayEquals(iArr, a4.getIntegerArray());
Assert.assertTrue("string".equals(a5.getName()));
Assert.assertEquals("a string", a5.getStringValue());
Assert.assertTrue("sArray".equals(a6.getName()));
Assert.assertArrayEquals(sArr, a6.getStringArray());
QA Software Test Plan
PROC-0015, Revision A Page 74 of 88
// Attribute value is turned into TRUE/FALSE string
Assert.assertTrue("bool".equals(a7.getName()));
Assert.assertTrue(Boolean.parseBoolean(a7.getStringValue()));
Assert.assertTrue("double".equals(a8.getName()));
Assert.assertEquals(null, Double.MAX_VALUE,
a8.getDouble().doubleValue(),
0.0d);
Assert.assertTrue("float".equals(a9.getName()));
Assert
.assertEquals(null, Float.MAX_VALUE,
a9.getFloat().floatValue(), 0.0d);
// No way of extracting primitives, so test using corresponding
classes
Assert.assertTrue("floatArray".equals(aA.getName()));
Float[] vA = aA.getFloatArray();
Assert.assertTrue(vA.length == f_Arr.length);
i = 0;
for (Float f : vA) {
Assert.assertEquals(null, f.floatValue(), f_Arr[i++],
0.0d);
}
Assert.assertTrue("intArray".equals(aB.getName()));
Integer[] vB = aB.getIntegerArray();
Assert.assertTrue(vB.length == i_Arr.length);
i = 0;
for (Integer i2 : vB) {
Assert.assertEquals(i2.intValue(), i_Arr[i++]);
}
Assert.assertTrue("long".equals(aC.getName()));
Assert.assertEquals(Long.MAX_VALUE, aC.getLong().longValue());
Assert.assertTrue("longArray".equals(aD.getName()));
Long[] vD = aD.getLongArray();
Assert.assertTrue(vD.length == l_Arr.length);
i = 0;
for (Long l : vD) {
Assert.assertEquals(l.longValue(), l_Arr[i++]);
}
}
}
8.3 EXAMPLE C++ UNIT TEST
The following example test is taken from the original CSF C++ unit tests and is out of date and may not
work. This example should be replaced once we have written some updated test classes.
#ifndef UT001_DATATESTSUITE_H_
#define UT001_DATATESTSUITE_H_
#include "cxxtest/TestSuite.h"
#include "cs/Attribute.h"
QA Software Test Plan
PROC-0015, Revision A Page 75 of 88
#include "cs/IAttribute.h"
#include "cs/AttributeTable.h"
#include "cs/Configuration.h"
#include "cs/HealthReport.h"
#include "cs/HealthRecord.h"
#include "cs/Health.h"
#include <fstream>
#include <iostream>
using namespace atst::cs::data;
using namespace atst::cs::interfaces;
using namespace atst::cs::util;
using atst::cs::services::Health;
class DataTestSuite : public CxxTest::TestSuite {
public:
void testAttribute ( void ) {
std::cout << std::endl;
// Test the creation of attributes
pIAttribute a1, a2, a3, a4, a5, a6, a7, a8;
pIAttribute a10, a11, a12, a13, a14, a15;
shared_ptr<Attribute> a9;
TS_ASSERT_THROWS_NOTHING(a1 = Attribute::create());
TS_ASSERT_THROWS_NOTHING(a2 = Attribute::create("test1"));
TS_ASSERT_THROWS_NOTHING(a3 = Attribute::create("test2", true));
TS_ASSERT_THROWS_NOTHING(a4 = Attribute::create("test3", 1));
TS_ASSERT_THROWS_NOTHING(a5 = Attribute::create("test4", (int64_t)2L));
TS_ASSERT_THROWS_NOTHING(a6 = Attribute::create("test5", (double)0.1));
TS_ASSERT_THROWS_NOTHING(a7 = Attribute::create("test6", (float)10.5));
TS_ASSERT_THROWS_NOTHING(a8 = Attribute::create("test7", "value"));
std::vector<bool> v1;
v1.push_back(false);
v1.push_back(true);
TS_ASSERT_THROWS_NOTHING(a9
= shared_ptr<Attribute>(new Attribute("test8", v1)));
std::vector<int> v2;
v2.push_back(3);
v2.push_back(5);
TS_ASSERT_THROWS_NOTHING(a10 = Attribute::create("test9", v2));
std::vector<int64_t> v3;
v3.push_back((int64_t)10);
v3.push_back((int64_t)20);
TS_ASSERT_THROWS_NOTHING(a11 = Attribute::create("test10", v3));
std::vector<double> v4;
v4.push_back(5.3);
v4.push_back(7.1);
TS_ASSERT_THROWS_NOTHING(a12 = Attribute::create("test11", v4));
std::vector<float> v5;
v5.push_back(9.2);
v5.push_back(8.5);
TS_ASSERT_THROWS_NOTHING(a13 = Attribute::create("test12", v5));
std::vector<std::string> v6;
v6.push_back("value1");
v6.push_back("value2");
TS_ASSERT_THROWS_NOTHING(a14 = Attribute::create("test13", v6));
QA Software Test Plan
PROC-0015, Revision A Page 76 of 88
std::string s1 = "abc";
std::string s2 = "defg";
TS_ASSERT_THROWS_NOTHING(a15 = Attribute::create("test14", s1));
// Test the checking of attribute values
TS_ASSERT(a3->getBoolean() == true);
TS_ASSERT(a4->getInteger() == 1);
TS_ASSERT(a5->getLong() == (int64_t)2L);
TS_ASSERT(a6->getDouble() == 0.1);
TS_ASSERT(a7->getFloat() == 10.5);
TS_ASSERT(a8->getString() == "value");
TS_ASSERT(a9->getBooleanArray() == v1);
TS_ASSERT(a10->getIntegerArray() == v2);
TS_ASSERT(a11->getLongArray() == v3);
TS_ASSERT(a12->getDoubleArray() == v4);
TS_ASSERT(a13->getFloatArray() == v5);
TS_ASSERT(a14->getStringArray() == v6);
TS_ASSERT(a15->getString() == "abc");
// Test the name setting and getting
TS_ASSERT_THROWS_NOTHING(a1->setName("test99"));
TS_ASSERT(a1->getName() == "test99");
// Test lack of a value
TS_ASSERT_THROWS_NOTHING(a1->isEmpty());
TS_ASSERT(a1->isEmpty() == true);
// Test the value setting
TS_ASSERT_THROWS_NOTHING(a3->setValue(false));
TS_ASSERT_THROWS_NOTHING(a4->setValue(5));
TS_ASSERT_THROWS_NOTHING(a5->setValue((int64_t)10L));
TS_ASSERT_THROWS_NOTHING(a6->setValue(0.5));
TS_ASSERT_THROWS_NOTHING(a7->setValue(2.5));
TS_ASSERT_THROWS_NOTHING(a8->setValue("value2"));
TS_ASSERT_THROWS_NOTHING(a9->setValue(v6));
TS_ASSERT_THROWS_NOTHING(a10->setValue(v5));
TS_ASSERT_THROWS_NOTHING(a11->setValue(v4));
TS_ASSERT_THROWS_NOTHING(a12->setValue(v3));
TS_ASSERT_THROWS_NOTHING(a13->setValue(v2));
TS_ASSERT_THROWS_NOTHING(a14->setValue(v1));
TS_ASSERT_THROWS_NOTHING(a15->setValue(s2));
// Recheck the attribute values
TS_ASSERT(a3->getBoolean() == false);
TS_ASSERT(a3->isEmpty() == false);
TS_ASSERT(a4->getInteger() == 5);
TS_ASSERT(a5->getLong() == (int64_t)10L);
TS_ASSERT(a6->getDouble() == 0.5);
TS_ASSERT(a7->getFloat() == 2.5);
TS_ASSERT(a8->getString() == "value2");
TS_ASSERT(a9->getStringArray() == v6);
TS_ASSERT(a10->getFloatArray() == v5);
TS_ASSERT(a11->getDoubleArray() == v4);
TS_ASSERT(a12->getLongArray() == v3);
TS_ASSERT(a13->getIntegerArray() == v2);
TS_ASSERT(a14->getBooleanArray() == v1);
TS_ASSERT(a15->getString() == "defg");
QA Software Test Plan
PROC-0015, Revision A Page 77 of 88
// Check the toString and show methods complete
TS_ASSERT(a13->toString() == "test12: 3, 5");
TS_ASSERT_THROWS_NOTHING(a14->show("Testing attribute->show()"));
pIAttribute a16;
TS_ASSERT_THROWS_NOTHING(a16 = a9->clone());
TS_ASSERT(a16->getName() == "test8");
TS_ASSERT(a16->getStringArray() == v6);
shared_ptr<Attribute> a_write;
TS_ASSERT_THROWS_NOTHING( a_write =
shared_ptr<Attribute>(new Attribute("name1",
"value1")));
std::ofstream outfile("writeData.out");
TS_ASSERT_THROWS_NOTHING( a_write->writeData( outfile ) );
std::ifstream infile("writeData.out");
std::tr1::shared_ptr<IAttribute> attr_in;
TS_ASSERT_THROWS_NOTHING( attr_in = Attribute::readData(infile));
TS_ASSERT(attr_in);
if (attr_in) {
TS_ASSERT( attr_in->getName() == "name1" );
TS_ASSERT( attr_in->getString() == "value1" );
}
pIAttribute attr_parse;
TS_ASSERT_THROWS_NOTHING( attr_parse =
Attribute::parseAttribute("name2\tvalue2") );
TS_ASSERT(attr_parse);
if (attr_parse) {
TS_ASSERT( attr_parse->getName() == "name2" );
TS_ASSERT( attr_parse->getString() == "value2" );
}
}
};
#endif /*UT001_DATATESTSUITE_H_*/
8.4 ADDING A UNIT TEST TO A TEST SET
To add the unit tests to a test set simply open the TAF configuration file for the test set that that this test
should be appended to (or multiple configuration files). Search for the "TESTS" section and then the
"UNIT" section. Add two new entries, one for the Java version and one for the C++ version.
<TESTS>
<UNIT>
<JAVA>test/java/unit_tests_001</JAVA>
<C++>test/c++/unit_tests_001</C++>
</UNIT>
</TESTS>
Individual unit test files can be added to as many TAF configuration files as required, there is no reason
not to run every unit test on every test run. Equally to keep certain test runs short it may not be necessary
to execute all unit tests.
QA Software Test Plan
PROC-0015, Revision A Page 78 of 88
9. SYSTEM TESTING
System testing is carried out on a fully integrated software system to verify that the system meets its
specified requirements. The system tests should have no knowledge of the inner design or logic of the
software that is being tested. The DKIST CSF provides a Jython interface to its public API which is ideal
for system tests as they can be written using the python language and kept away from the internal DKIST
codebase
9.1 WRITING A PYTHON SYSTEM TEST
A system test script is simply a Python file that subclasses the relevant testing classes described in section
4.3. This section provides a step by step method for writing a system test. This example will use the case
of posting and receiving an event. The full listing of this test can be found in section 9.2.
1) Identify the name of the new system test. There is no restriction placed on the naming of tests but if
the name includes a short description of what the test covers it is easier to manage large sets of tests. For
this example the test name is ATST_SYS_001_EventPost.
2) Create a new python file under [TBD] the name of which is the same as decided in step 1
(ATST_SYS_001_EventPost.py).
3) Some imports are required at the top of the file. These imports expose the API required for writing
system tests, all test scripts should start with these imports.
import sys
if sys.path.count(Misc.getRelease() + 'src/java/atst/tcs/test') == 0:
sys.path.append(Misc.getRelease() + 'src/java/atst/tcs/test')
from TestFrameworkFailure import TestFrameworkFailure
from TestFrameworkStep import TestFrameworkStep
from TestFramework import TestFramework
4) Additional imports for CSF specific functionality may be desired. The full CSF service set can be
imported.
from atst.cs.util import *
from atst.cs.services import *
from atst.cs.services.event import *
4) For each test step a new python class must be created that subclasses the TestFrameworkClass. It is a
good idea to use the name of the test and append the test step number to it, to ensure classes inside each
test script cannot inadvertently overwrite other scripts.
class ATST_SYS_001_EventPost_Step01( TestFrameworkClass ):
5) Implement the run method of the class. There is no restriction placed on what occurs inside the run
method but making use of the provided API defined in section 4.3 will make test scripts consistent and
easier to manage in the future.
def Run( self ):
6) In the case of our test we want to post an event and then listen for that event. To be able to do this we
need to post asynchronously using the PostAsync call with a delay of 1.0 seconds.
QA Software Test Plan
PROC-0015, Revision A Page 79 of 88
table = AttributeTable()
table.insert(Attribute("testItem", "true"))
self.PostAsync( "atst.testChannel", table, 1.0)
7) Now let's log a message to state the table has been posted and then wait to receive the event. The
Listen call will block until the event is received.
self.LogMessage("Test event has been posted")
response = self.Listen("atst.testChannel", [".testItem"])
8) Finally let's log the contents of the response and fail if the value is not true.
self.LogMessage("Received event: " + response.toString())
if (response.getString(".testItem") != "true"):
self.Fail("The testItem value was not true as expected")
9) Once the test steps have been completed it is necessary to register the test with the framework so that
the test executor can pick up and execute the script when necessary. First create a new test object.
ATST_SYS_001_EventPost = Test("ATST_SYS_001_EventPost")
10) Provide a description and add the test step.
ATST_SYS_001_EventPost.setDescription("Testing event posting and receiving.")
ATST_SYS_001_EventPost.addStep(ATST_SYS_001_EventPost_Step01)
11) Register the test with the framework
TestFramework.Register(ATST_SYS_001_EventPost)
12) The test script is now complete and ready for execution. The full script is presented in section 9.2
below.
9.2 EXAMPLE PYTHON SYSTEM TEST
The following example is a simple system test created for the purposes of this document. This example
should be updated with additional steps as the test framework software evolves.
import sys
if sys.path.count(Misc.getRelease() + 'src/java/atst/tcs/test') == 0:
sys.path.append(Misc.getRelease() + 'src/java/atst/tcs/test')
from TestFrameworkFailure import TestFrameworkFailure
from TestFrameworkStep import TestFrameworkStep
from TestFramework import TestFramework
from atst.cs.util import *
from atst.cs.services import *
from atst.cs.services.event import *
class ATST_SYS_001_EventPost_Step01( TestFrameworkClass ):
def Run( self ):
table = AttributeTable()
table.insert(Attribute("testItem", "true"))
self.PostAsync( "atst.testChannel", table, 1.0)
self.LogMessage("Test event has been posted")
response = self.Listen("atst.testChannel", [".testItem"])
QA Software Test Plan
PROC-0015, Revision A Page 80 of 88
self.LogMessage("Received event: " + response.toString())
if (response.getString(".testItem") != "true"):
self.Fail("The testItem value was not true as expected")
ATST_SYS_001_EventPost = Test("ATST_SYS_001_EventPost")
ATST_SYS_001_EventPost.setDescription("Testing event posting and receiving.")
ATST_SYS_001_EventPost.addStep(ATST_SYS_001_EventPost_Step01)
TestFramework.Register(ATST_SYS_001_EventPost)
9.3 ADDING A PYTHON SYSTEM TEST TO A TEST SET
To add system tests to a test set simply open the TAF configuration file for the test set that that this test
should be appended to (or multiple configuration files). Search for the "TESTS" section and then the
"SYSTEM" section. Add a new entry for the Python test.
<TESTS>
<SYSTEM>
<PYTHON>test/python/ATST_SYS_001_EventPost</PYTHON>
</SYSTEM>
</TESTS>
Individual system test files can be added to as many TAF configuration files as required, there is no
reason not to run every system test on every test run. Equally to keep certain test runs short it may not be
necessary to execute all system tests.
QA Software Test Plan
PROC-0015, Revision A Page 81 of 88
10. REGRESSION TESTING
Regression testing is used to determine any new bugs that have been introduced into a system after a new
release has been made. As part of the testing procedure any new bug that is found within DKIST
software should have a test written (unit, system or a combination) that firstly can be used to reproduce
the fault, and secondly can be used to verify the fault has been fixed. From the point at which the fault
has been verified as fixed the test becomes part of the regression test suite, and should continue to be
executed on every new release of the software to ensure the fault has not re-occurred. By definition every
DKIST test that is written to demonstrate a fault will become part of the regression test suite for future
releases.
By utilizing the Testing Automation Framework (section 5) and the E2E Test Executor (section 6)
regression testing should be a simple case of adding the test file to the list of defined tests for DKIST and
it will then be executed automatically when these frameworks are run.
10.1 SETTING UP A REGRESSION TEST RUN
To setup a regression test run the tester should first determine which tests are to be run. This may range
from a small set of tests looking for a specific problem up to every test that has been written for the
specific system that is undergoing regression tests. If we use the example of CSF then it is quite likely
that the list of tests will include all available unit tests and CSF system tests.
Once the tests have been decided upon a TAF configuration file is created to setup the test run (see
section 5.3). For the case of CSF the configuration file would define the version of CSF to checkout only,
no other software modules should be included. The test scripts are then added along with the desired
location for test reports. An example configuration file is presented below:
<TAF>
<ID>Regression 01</ID>
<REPORT>/mount/reports/regression</REPORT>
<SOFTWARE>
<CSF>Canary_5</CSF>
</SOFTWARE>
<TESTS>
<UNIT>
<JAVA>UnitTest1</JAVA>
<JAVA>UnitTest2</JAVA>
<C++>UnitTest1</C++>
<C++>UnitTest2</C++>
</UNIT>
<SYSTEM>
<PYTHON>SystemTest1</PYTHON>
<PYTHON>SystemTest2</PYTHON>
</SYSTEM>
</TESTS>
</TAF>
Once the TAF configuration file has been created the tester can run the regression tests immediately by
simply executing the TAF and passing the configuration file to it. If it is preferable to automate the
execution of the regression tests, perhaps to schedule the execution for the same time each day then the
file can be added to the ETE (see section 6.3).
QA Software Test Plan
PROC-0015, Revision A Page 82 of 88
11. ETE TEST SERVER
An ETE environment has been set up on an ETE server schwabe. Currently there is one VMWare
workstation virtual machine under the home directory of username ‘qas’. It has been set up to checkout
the trunk version of csf, base and qas modules and run system tests and unit tests every day.
Path of the VMware workstation client: /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx
There is a snapshot ETEReady. The snapshot was taken when the virtual machine was set up so that it is
ready to install atst csf and base modules of version Canary_8.
The atst_test module is checked out to /home/qas/ATST_TEST/atst_test.
The directory for the TAF report directory, where the report for each test script will be kept, is
/home/qas/ATST_TEST/TAFreports. This directory is shared with the virtual client. On the virtual client
machine, the path for this directory is /mnt/hgfs/Reports. Figure shows the setting of the shared folder of
the virtual machine.
Figure 2 Shared folder setting
The cron job for the ETE is set up as below.
QA Software Test Plan
PROC-0015, Revision A Page 83 of 88
*/30 * * * * export ATST_TEST=/home/qas/ATST_TEST/atst_test;filename=`eval date -u
+\%Y\%m\%d_\%H\%M\%S`"_ETE.log";/home/qas/ATST_TEST/atst_test/bin/ETE --
eteconfig=/home/qas/ATST_TEST/configs/ETE_config_centos66.xml --vmuser=atst-qas --
vmpassword=<password> --emailpgm=mail > /home/qas/ATST_TEST/ETElogs/$filename
The ETE is called every 30 minutes. The ETE configure file is
/home/qas/ATST_TEST/configs/ETE_config_centos66.xml and it is set as shown below. The test will be
executed every day at 09:00 UTC and the result email will be sent.
<ETE>
<instance time="09:00">
<EXECUTION>
<VM>
<VM_FILE>/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx</VM_FILE>
<TAF_CONFIG>/home/qas/ATST_TEST/configs/TAF_config_HEAD_Install_and_test.xml</TAF_CONF
IG>
<REPORT_DIR_PATH_ON_VM>/mnt/hgfs/Reports</REPORT_DIR_PATH_ON_VM>
<REPORT_DIR_PATH_ON_HOST>/home/qas/ATST_TEST/TAFreports</REPORT_DIR_PATH_ON_HOS
T>
<SNAPSHOT>ETEready</SNAPSHOT>
<SYSTEM_TESTS>1-51,56,58,73-77,79-94</SYSTEM_TESTS>
<JAVA_UNIT_TESTS>10-15</JAVA_UNIT_TESTS>
<CPP_UNIT_TESTS>7,10-16</CPP_UNIT_TESTS>
<RUN_ORDER>1</RUN_ORDER>
</VM>
</EXECUTION>
<EMAIL_ADDRESS>aya@observatorysciences.co.uk</EMAIL_ADDRESS>
<FINAL_REPORT_DIR>/home/qas/ATST_TEST/ETEFinalReports</FINAL_REPORT_DIR>
</instance>
</ETE>
The TAF configure file is also stored under /home/qas/ATST_TEST/configs directory.
Below is an example of email sent from the ETE. The email subject will include [FAIL] if there are any
failed tests.
Subject: Test report for ETE_config_centos66.xml [FAIL]
Test Execution: Fri Feb 20 09:00:01 2015
Virtual machine: /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx
TAF config file:
/home/qas/ATST_TEST/configs/TAF_config_HEAD_Install_and_test.xml
Results Summary:
Run Pass Fail
Unit 14 12 2
System 74 73 1
Details of failures:
ID = SYS_REQ_4.1_0101
CVS ID = $Id: SYS_REQ_4.1_0101.py,v 1.5 2014/09/10 09:20:43 ChrisMayer Exp $
VM name = /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx
Snapshot = ETEready
TAF ID = Installing HEAD and test
Type = system
Time = 2015-02-20 09:09:25.260
Pass = False
Machine = qas01
QA Software Test Plan
PROC-0015, Revision A Page 84 of 88
Failures =
Failure 1
Reason = The container SYSREQ410101JAVA2 is not registered.
Step = 7
Location = StepCheckContainerIsRegistered
Versions =
[CSF=HEAD]
[Base=HEAD]
ID = REG_CANARY_8_ATCS_791.h
CVS ID = $Id: REG_CANARY_8_ATCS_791.h,v 1.1 2014/10/01 09:03:09 ChrisMayer Exp $
VM name = "/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx"
Snapshot = "ETEready"
TAF ID = "Installing
Type = unit
Time = 2015/02/20:10:37:24.247 GMT
Pass = False
Machine = qas01
Failures =
Locations =
REG_CANARY_8_ATCS_791.h[140]
Reasons =
Error: Expected (nanoStringOut[i]+gmt == d1->toNanoString()), found
("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")
Versions =
[CSF=HEAD]
[Base=HEAD]
ID = REG_CANARY_8_ATCS_792.h
CVS ID = $Id: REG_CANARY_8_ATCS_792.h,v 1.1 2014/10/01 09:03:18 ChrisMayer Exp $
VM name = "/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx"
Snapshot = "ETEready"
TAF ID = "Installing
Type = unit
Time = 2015/02/20:10:37:29.381 GMT
Pass = False
Machine = qas01
Failures =
Locations =
REG_CANARY_8_ATCS_792.h[92]
Reasons =
Error: Expected (nanoStringOut[i] == d1->toNanoString()), found
("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")
Versions =
[CSF=HEAD]
[Base=HEAD]
--------------End of report--------------
To run another test, for example testing another versions of csf and base, just add a new test instance to
the ETE configure file. One thing to be careful of is the execution time. Because the cron job is run every
30 minutes, the test execution time should be set to 0 or 30 minutes past the hour. Another thing to note is
that the ETE can’t run tests if the virtual machine is running. For example, the test at 9:00 UTC takes
about one and half hours to complete. The new test’s time should be set to 11:00 UTC or later if the same
virtual machine is used.
To edit configure files on schwabe, log in to the server as a user within the qas group. The new TAF
configure file can be created using configure_taf or configure_tests tools (see 5.4.1). After logging in to
schwabe, make sure to set the environment variable ATST_TEST to /home/qas/ATST_TEST/atst_test.
QA Software Test Plan
PROC-0015, Revision A Page 85 of 88
The default file taf_defaults for the TAF configure file tools has been set up for the virtual machine
QAS01_CentOS66.vmx. Part of the file is shown below.
INST_DIR=/opt
REPORT=/mnt/hgfs/Reports
CREATE_DEVEL_ACTION=--make-all
MAKE_ACTION=build_all
POSTGRESQL=postgresql-9.3
INITDB=Yes
ICESERVICE=Yes
INSTALL_APP=Yes
#Software defaults should be placed between 'SOFTWARES_START' and
'SOFTWARES_END'
#It should be specified as <Software name>=<Version>
SOFTWARES_START
CSF=HEAD
BASE=HEAD
QAS=HEAD
SOFTWARES_END
#Site config default items should be placed between 'SITE_CONFIG_START' and
'SITE_CONFIG_END'
SITE_CONFIG_START
baseDir=$ATST
ATST_RELEASE_FLAG=trunk
ATST_RUN_ENVIRON=/opt/atst/var
ATST_JAVA_HOME=/opt/java8
ATST_JLIB_DIRS=none
USE_CPP=yes
ATST_CLIB_DIRS=/usr/pgsql-9.3/lib/:/usr/local/lib/
ATST_ARCHIVE_DB_HOST=localhost
ATST_ID_DB_HOST=localhost
ATST_LOG_DB_HOST=localhost
ATST_PROPERTY_DB_HOST=localhost
ATST_PARAM_SET_DB_HOST=localhost
ATST_SCRIPT_DB_HOST=localhost
ATST_HEADER_DB_HOST=localhost
ATST_ICE_VERSION=3.5.1
ATST_ICE_CONNECTION_HOST=localhost
ATST_ICE_EVENTS_HOST=localhost
ATST_ALARM_DB_HOST=localhost
ATST_BULK_DATA_DB_HOST=localhost
SITE_CONFIG_END
…
The ETE configure file on schwabe can be edited using configure_ete or configure_tests tools (see 6.3.2).
The default file ete_defaults for the ETE configure file tools has been set up for the ETE server.
[aya@schwabe bin]$ more ../src/python/editor/ete_defaults
VM_FILE=/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx
#TAF_CONFIG=
REPORT_DIR_PATH_ON_VM=/mnt/hgfs/Reports
REPORT_DIR_PATH_ON_HOST=/home/qas/ATST_TEST/TAFreports
FINAL_REPORT_DIR=/home/qas/ATST_TEST/ETEFinalReports
QA Software Test Plan
PROC-0015, Revision A Page 86 of 88
#EMAIL_ADDRESS=
#SNAPSHOT=
#SYSTEM_TESTS=
#JAVA_UNIT_TESTS=
#CPP_UNIT_TESTS=
QA Software Test Plan
PROC-0015, Revision A Page 87 of 88
12. FUTURE UPDATES
Below is a list of possible future updates.
1. The TAF should be able to install atst using pkgDevel.
2. The TAF should be able to do ‘cvs update’. Currently it can only do ‘cvs checkout’.
3. The TAF should be able to checkout any modules specified in a TAF configuration file. Currently
it can only checkout CSF, BASE, TCS and QAS.
4. The TAF should be updated to be able to take a snapshot of a virtual machine in case of test
failure.
5. TestFrameworkStep.py should implement a method to submit a configuration with timeout.
QA Software Test Plan
PROC-0015, Revision A Page 88 of 88
13. REFERENCES
[1] Wampler, W. & Goodrich, B., ATST Software Design Document, SPEC-0014, Rev. 0.3
[2] Greer, A. & Mayer, C., ATST TCS Acceptance Test Plan, SPEC-xxxx, Rev. A
[3] Kneale, R., ATST Glossary and Acronym List, SPEC-0012, Rev. D1
[4] Wampler, S. & Goodrich, B., ATST Common Services Software Design Document Vol. I User
Manual, SPEC-0022-01, Rev. A4
[5] Greer, A., Hubbard, J. & Wampler, S., Java Engineering Screens User Manual, TN-0089, Rev.
3.0/A11/A
top related