qt test framework
TRANSCRIPT
QtTest
Unit Testing Framework
Justin Noel
Senior Consulting Engineer
ICS, Inc.
Qt is the Kitchen Sink Toolkit
• Qt comes with just about everything you need
• SQL, Sockets, XML, GUI, etc
• Often one of the challenges of using Qt is knowing all the
modules
• And what they do!
• This is ICS performs these webinars!
• Qt comes with it’s own Unit Testing Framework
• QtTest
Types of Testing
• Unit Testing
• Test one discrete area of the software
• Usually on a class basis. Think “Test Piston Rod”
• Tests can be written and performed as code is developed
• White box tests. Often written by developers
• “Does the software perform as the developer intended?”
• Functional Testing
• Tests subsystems of the whole software system
• Usually a group of classes. Think “Test Engine”
• Tests can be preformed once subsystem is wired together
• Tests interaction between specific groups of units
• “Does the software work together?”
Types of Testing
• System Testing
• End to end testing of the whole software system
• Sometimes includes actual hardware. Think “Test Car”.
• Tests are written late in project
• Because of the end to end nature.
• Black box tests. Often written by separate SQA team
• “Does the software do right thing?”
Test Frameworks for Qt
• QTest / QTest-QML
• Unit testing framework that comes with Qt
• Qt specific functionality
• Can test blind signals. Can simulate mouse and key events.
• Qt specific features can be used outside of QTest framework
• Google Test
• Unit testing framework from Google
• Unique ability to do “Isolated Testing” via Google Mock
• Requires “Inversion of Control” pattern
• Squish
• Playback / Record frame work for “end to end” system testing.
• Used for System Testing
Benefits of Unit Testing
• Test Early!
• Find errors while code is actively being developed
• Rather than long investigations during system testing
• Test Often!
• Run unit tests per commit. Or even pre-commit!
• Any failures must have been introduced by this 200 lines of code!
• Test for Regressions!
• Did new functionality or bug fixing break existing functionality?
• What tests failed?
• Did a code change re-introduce a bug?
• Every bug can have it’s own set of unit tests.
Design for Testability
• Your code must be testable!
• There are design techniques to help make your code more testable
• Try not to add functions to production code just for tests
• Subclassing for tests is acceptable (.ui file members)
• Isolation of units is your primary goal
• Can I test just this class?
• Without re-testing lower level classes?
• Also think about error conditions
• Some may be hard to produce in production environments
• Can I stimulate a bad MD5 error?
• Socket disconnects?
• Default switch cases?
Build Application as Libraries
• Building the bulk of your application as a set of libraries
increases testability
AppBackend.dll
App.exe
(main.cpp)
AppUi.dll
BackendTests.exe UiTests.exe
C++ Test Fixtures QML Test Fixtures
Layered Design
• A layered design is more testable
• Easy to isolate lower layers for testing
• Test Communications without Data, Presentation or Visualization
• No upwards dependencies
• Hard downward dependencies
Visualization Layer (QML)
Presentation Layer (QObjects)
Data Layer (Local Data Storage)
Communications Layer (TCP)
Calls
Down
Signals
Up
Break Downward Dependencies
• Dependency injection works well
• An Inversion of Control Pattern
• Classes take their dependencies as constructor arguments
• Classes do not construct any members themselves
• Loose coupling with signals and slots also works well
• Classes interface to each other via signals and slots only
• 3rd class wires say the UI classes to the Backend classes
• Lets you substitute a test fixture object for a dependency
object
• Instead of an actual production object.
Why is isolation important?
• Avoids testing the same code over and over
• You have already tested the data model
• Why do the UI tests need to also test the model?
• This will make your tests run very quickly
• If there is a failure in a low level class it won’t trigger errors in higher
level tests
• Avoid testing side effects
• Tests can be implementation agnostic
• DB, Files, Cloud. Shouldn’t matter for UI tests.
• Tests should only depend on interfaces, not behavior
EE Analogy: Motor Controller Test
Motor Controller
7408 AND
Momentary
Switch
Safety
Interlock
Motor
1
2
3
Stimulus: Apply PowerVerify: High or Low
QtTest is a unit testing framework
• QtTest comes with classes to build test fixtures for units
• Has some “special sauce” for Qt
• Test signals without needing to write slots (QSignalSpy)
• Simulate user events (MousePress, KeyPress, FocusIn, etc)
• QtTest is not an isolation framework
• You will have to make replacement classes for dependencies
yourself
• These are usually referred to as Mock Classes.
• Next month we will talk about Google Mock which is can help a lot
with isolation framework.
Creating a Test Fixture
• Can be implemented inside a source file
• No need for a header file
• Inherit from QObject
• Qt uses introspection to deduce test functions
• Private slots are test functions
• There are some reserved function names
• Use Q_TEST_MAIN macro to generate a main()
• Or create you own main.cpp to run multiple test fixtures
Creating a Test Fixture
#include <QtTest>
class TestQString: public QObject
{
Q_OBJECT
private slots:
void toUpper() {
QString str = "Hello"; // Prepare
str = str.toUpper(); // Stimulate
QCOMPARE(str, QString("HELLO")); // Verify
}
};
QTEST_MAIN(TestQString)
#include “testqstring.moc”
Test Fixture Project
QT += testlib # Plus libs your uses.
CONFIG += testcase # Creates make check target
SOURCES += TestQString.cpp
qmake
make
make check
********* Start testing of TestQString *********
Config: Using QtTest library %VERSION%, Qt %VERSION%
PASS : TestQString::initTestCase()
PASS : TestQString::toUpper()
PASS : TestQString::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of TestQString *********
Running Multiple Tests
#include “QTestString.h” // Easier if you use a header file
int main(int argc, char** argc)
{
bool success = true;
r &= QTest::qExec(TestQString(), argc, argv);
...
return success;
}
Initializing and Cleaning Up
class TestQString: public QObject
{
Q_OBJECT
private slots:
void initTestCase() { // Called before ALL tests. }
void cleanupTestCase() { // Called after ALL tests}
void init() { // Called before each test. }
void cleanup() { // Called after each test }
void toUpper();
void toLower();
};
Writing Good Tests
• Test one thing at a time
• It’s tempting to get as much done in one function as you can.
• This makes one verification depend on another verification
• Tests will be brittle and changes to code could break lots of
verifications
• Do not have your tests depend on each other
• Order should not matter!
• Not depending on other tests will help you when you remove
functionality.
• You do not want to have to fix a cascading waterfall of tests!
Use New Object For Each Test
• Use init()
• Create your Object Under Test
• Plus whatever dependencies it has
• Production dependencies
• Or Test dependencies like Mock Classes
• Use cleanup()
• Do not be sloppy and leak memory in you tests.
• Your tests should run clean under Valgrind
• Aside for some suppressions for FreeType and FontConfig
Use New Object For Each Test
class MotorControllerTests: public QObject
{
Q_OBJECT
private slots:
void init() { controller = new MotorController; }
void cleanup() { delete controller; }
void motorRunsIfInterlockAndSwitchAreTrue() {
controller.setInterLock(true);
controller.setSwitch(true);
QCOMPARE(controller.motorOn(), true)
}
void motorDoesNotRunIfInterlockIsFalse() { ... }
private:
MotorController* controller;
};
Data Driven Tests
• Sometimes you need to test many inputs to a function
• NULL, Same Value Twice, Negative, Positive, NAN, etc
• This isn’t great test code. Violates Test 1 ThingQCOMPARE(QString("hello").toUpper(), QString("HELLO"));
QCOMPARE(QString("Hello").toUpper(), QString("HELLO"));
QCOMPARE(QString("HellO").toUpper(), QString("HELLO"));
QCOMPARE(QString("HELLO").toUpper(), QString("HELLO"));
• Or worse this. Lots of error prone typing if not trivial.void testhelloToUpper() {...}
void testHELLOToUpper() {...}
void testHeLOToUpper() {...}
Data Driven Tests
private slots:
void toUpper_data() {
QTest::addColumn<QString>("string");
QTest::addColumn<QString>("result");
//We can name the sub tests
QTest::newRow("all lower") << "hello" << "HELLO";
QTest::newRow("mixed") << "Hello" << "HELLO";
QTest::newRow("all upper") << "HELLO" << "HELLO";
}
void TestQString::toUpper() { // Called 3 Times
QFETCH(QString, string);
QFETCH(QString, result);
QCOMPARE(string.toUpper(), result);
}
};
Testing Events and Signals
• Use QSignalSpy to attach to any signal
• Signal parameters need to be QVariant compatible
• May require Q_DECLARE_METATYPE()
• May require qRegisterMetaType<>()
• Verify with count() and arguments()
• Use QtTest static methods to stimulate events
• void mousePress(widget, button, modifiers, pos)
• void keyClicks(widget, string, modifiers)
• Etc, etc
Testing Events and Signals
class QPushButtonTests: public QObject
{
Q_OBJECT
private slots:
void init() { //Create, Show, Wait For Window Shown }
void emitsClickedWhenLeftMouseButtonIsClicked() {
QSignalSpy spy(button, &QPushButton::clicked);
QTest::mouseClick(button, Qt::LeftButton)
QCOMPARE(spy.count(), 1);
}
private:
QPushButton* button;
};
Stimulating Signals
• Often your object under test is depending on signals from
other objects.
• For example QPushButton clicked()
• Don’t re-test the button
• Directly call QPushButton clicked()
• signals access specifier is actually a #define public
Stimulating Signals
private slots:
void init() {
settings = new Settings;
settingsDialog = new SettingsDialog(settings);
}
void LanguageComboBoxUpdatesOnSettingsLanguageChanged() {
QComboBox* combo = settings->ui()->langCombo;
settings->lanuageChanged(“en_US”);
QCOMPARE(combo->currentText(), “English”);
}
private:
Settings* settings;
TestableSettingsDialog* button; // Exposes ui member via function
};
Testing QML with QTest
• Writing tests in QML is very similar to C++
• Create a test fixture class
• Implement specially named functions
• Prepare, Stimulate, Verify.
• KISS – Keep It Short and Simple
• There are QML equivalents to
• compare, verify, wait
• SignalSpy
• mouseClick(...)
• TestCase
• Specific test fixture class. C++ just uses QObject.
UI Control Test
• UI Controls are easy to test as they have no dependencies
TestCase {
name: "Button“
Button { id: button }
SignalSpy { id: clickedSpy; target: button; signal: ”clicked” }
function test_mouseClickEmitClicked() //Tests are run in alpha order
{
mouseClick(button)
compare(clickedSpy.count, 1)
}
}
New Objects For Each Test
TestCase {
id: testCase
property Button button: null
name: "Button“
Component { id: buttonFactory; Button{} }
function init() { button = buttonFacotry.createObject(testCase) }
function cleanup() { button.destroy() }
function test_mouseClickEmitClicked()
{
mouseClick(button)
compare(clickedSpy.count, 1)
}
}
Running QML Tests
• qmltestrunner
• Prebuilt test runner.
• Takes a filename or directory of files to run
• Many command line options
• See Qt Documentation
• Can run many test by itself.
• All UI controls should be testable this way
• Custom main.cpp runner
• Reuse the guts of qmltestrunner
• Add injected type information or other setup
• As required by your app
qmltestrunner
$QTDIR/bin/qmltestrunner –input TestDir
Custom main.cpp
#include <QtQuickTest/quicktest.h>
#include <QtCore/qstring.h>
#ifdef QT_OPENGL_LIB
#include <QtOpenGL/qgl.h>
#endif
#define TEST_DIR “UiTests” // Pass this as DEFINE in project file
int main(int argc, char **argv)
{
//Register Types Here
return quick_test_main(argc, argv, “UiTests", TEST_DIR);
}
Screen Tests
• Screens are more interesting
• They depend on Controllers and Models from C++
• QML and C++ is natively a layered architecture
• Most common ways to communicate with C++ is injection
• Of C++ instances or C++ types
• However we can Mock Controllers and Models
• Using QML instead of C++
C++ Integration Mechanisms
• Context Properties – Injects global pointers at root context
• view.rootContext()->setContextProperty(“coffeeMaker”, &maker);
• QML Singletons – Inject a singleton factory for import
• qmlRegisterSingletonType<CoffeeMaker>(“MrCoffee", 1, 0,
“CoffeeMaker",
factoryFunction);
• Registered Types – Add new types for import
• qmlRegisterType<CoffeeMaker>(“MrCoffee“, 1, 0, “CoffeeMaker“);
Mocking Context Properties
• Create a Mock[ClassName].qml file
• Create the equivalent property, signal and function API
• The TestCase item is the root item for your test
• It literally is the rootContext()
• Simply add properties to the TestCase with the same names
• They will have global scope
• Tests can be run through the standard qmltestrunner
Mocking Singletons
• Create a Mock[ClassName].qml file
• Create the equivalent property, signal and function API
• Create a custom QML Test main.cpp
• Call qmlRegisterSingletonType with URL to file name of mock
• qmlRegisterSingletonType(QUrl("file:///path/MockCoffeeMaker.qml"),
“MrCoffee", 1, 0,
“CoffeeMaker");
• Run tests as usual.
• Objects will use mock versions of C++ classes
Mocking Registered Types
• Create a Mock[ClassName].qml file
• Create the equivalent property, signal and function API
• Inside a directory structure similar to production
• Mocks/com/ics/qtest/examples/
• Create a qmldir manifest file listing your mock types
• CoffeeMaker 1.0 MockCoffeeMaker.qml
• Manipulate QML2_IMPORT_PATH to pull in testing
module rather than production module
CoffeeMaker.h
class CoffeeMaker : public QObject
{
Q_OBJECT
Q_PROPERTY(int temp READ temp
NOTIFY tempChanged)
Q_PROPERTY(int targetTemp READ targetTemp
WRITE setTargetTemp
NOTIFY targetTempChanged)
public:
enum Strength {
Bold=0, Medium, Light
} Q_ENUM(Strength)
CoffeeMaker(IPump& pump, IHeater& heater);
...
Q_INVOKABLE void brew(Strength strength);
};
main.cppint main(int argc, char** argv)
{
QGuiApplication app(argc, argv);
...
CoffeeMaker coffeeMaker(pump, heater);
qmlRegisterUncreatableType<CoffeeMaker>(“MrCoffee”,
1, 0,
“CoffeeMaker”, “”);
QQuickView view;
view.rootContext()->setContextProperty(“coffeeMaker”, & coffeeMaker);
view.setSource(“qrc:/Main.qml”);
view.show();
return app.exec();
}
CoffeeScreen.qml
import QtQuick 2.5
import MyComponents 1.0
import MrCoffee 1.0
Screen {
...
SpinBox {
value: coffeeMaker.targetTemp
onValueChanged: coffeeMaker.targetTemp = value
Connections {
target: coffeeMaker
onTargetTempChanged: value = coffeeMaker.targetTemp
}
}
Button {
text: qsTr(“Brew”)
onClicked: coffeeMaker.brew(CoffeeMaker.Medium)
}
}
CoffeeScreen.qml With Hooks
Screen {
property alias _spinBox: spinBox // No QML protected specifier
property alias _button: button
...
SpinBox {
id: spinBox
value: coffeeMaker.targetTemp
onValueChanged: coffeeMaker.targetTemp = value
Connections {
target: coffeeMaker
onTargetTempChanged: value = coffeeMaker.targetTemp
}
}
Button {
id: button
text: qsTr(“Brew”)
onClicked: coffeeMaker.brew(CoffeeMaker.Medium)
}
}
Only Test CoffeeScreen
• CoffeeMaker is tested separately
• Is a discrete unit
• Has it’s own unit tests.
• A broken CoffeeMaker object shouldn’t fail CoffeeScreen tests
• UI Controls are tested separately
• SpinBox and Button are also discrete units
• Have their own tests
• Changing UI control APIs could cause tests not to run
• Controls are leaf nodes in the software design
• Easy to test by themselves.
• Should stabilize quickly
• When was the last time the API for SpinBox changed?
CoffeeScreen Text Fixture
CoffeeScreen
SpinBox
PushButton
MockCoffeeMaker
Stimulus: Set Values / EmitVerify: Was Function Called?
MockCoffeeMaker.qml
QtObject {
property int temp: 0
property int targetTemp: 0
property int brewInvokations: 0
property var brewStrengthParameter: []
function brew(strength) {
brewStrengthParameter[brewInvokations] = strength
brewInvokations++
}
function reset() {
brewStrengthParameter = []
brewInvokations = 0
}
}
TestCase with Mocks
UiTestCase.qml
---------------------------------------------------------------------
TestCase {
when: windowShown
property MockCoffeeMaker coffeeMaker: MockCoffeeMaker {}
function resetAllMocks() {
coffeeMaker.reset();
}
}
CoffeeScreenTest.qml
import “../qmlSource”
UiTestCase {
id: testCase
name: "CoffeeScreen“
property CoffeeScreen coffeeScreen: null
Component { id: factory; CoffeeScreen {} }
function init() {
resetAllMocks();
coffeeScreen = factory.createObject(testCase)
}
function cleanup() {
coffeeScreen.destroy()
coffeeScreen = null
}
...
CoffeeScreenTest.qml
...
function test_spinBoxValueIsBoundToCoffeeMakerTemp() {
coffeeMaker.temp = 50
compare(coffeeScreen._spinBox.value, 50)
}
}
// Bug #532
function test_settingSpinBoxValueDoesNotBreakBinding() {
coffeeScreen._spinBox.value = 99
coffeeMaker.temp = 20
compare(coffeeScreen._spinBox.value, 20)
}
}
CoffeeScreenTest.qml
...
function test_buttonClickedCallsBrew() {
coffeeScreen._button.clicked()
compare(coffeeMaker.brewInvokations, 1)
}
function test_buttonClickedCallsBrewWithMediumStregth() {
coffeeScreen._button.clicked()
compare(coffeeMaker.brewStrengthArgument[0], CoffeeMaker.Medium)
}
Custom Test Runner
#include <QtQuickTest/quicktest.h>
#include <QtCore/qstring.h>
#ifdef QT_OPENGL_LIB
#include <QtOpenGL/qgl.h>
#endif
#include “CoffeeMaker.h”
int main(int argc, char **argv)
{
qmlRegisterUncreatableType<CoffeeMaker>(“MrCoffee”, 1, 0,
“CoffeeMaker”, “”);
return quick_test_main(argc, argv, “UiTests", “.”);
}
Thank You!
Justin Noel
Senior Consulting Engineer
ICS, Inc.
www.ics.com