Definitions

Tests that involve contrived or instrumented example code that are written specially for the test:

• Unit Testing - testing the algorithmic properties of an item.
• Visual Testing - verifying that the visual output is correct.
• Performance Testing - collecting statistics on how fast a particular piece of code can be executed, including comparison against alternative implementations.
• Coverage Testing - unit testing, with tools to report the percentage of library, plugin, or application code covered.

Tests that involve actual end-user applications, with little formal instrumentation:

• System Testing - sending key clicks and mouse presses, and verifying the results against an application specification.

Existing test solutions for QML

• QmlUnit - based on QUnit, tests written in JavaScript, community project.
• Qt3D.Test - some concepts from qmlunit, simple algorithmic unit testing only. Consists of a small cpp test harness that launches scripts in a QDeclarativeView, with most of the test infrastructure written in QML and JavaScript. Has JavaScript equivalents of QVERIFY and QCOMPARE. Text and XML output modes.
• qmlvisual - visual testing tool: qmlviewer is used to record test scripts (in QML syntax) that are replayed and verified later.
• QTestLib for QtDeclarative - used to write tests for QtDeclarative to directly exercise the core elements from cpp. These tests are otherwise plain cpp Qt unit tests. Some tests load QML into a QDeclarativeEngine, but QVERIFY and QCOMPARE calls to check the results are still done in cpp.
• QtUiTest - plugin for creator for supporting system testing of cpp applications. Some very early support for QML, using the same testing style as for cpp tests. API tries to distance the user from the details of the underlying API elements, to make it more suitable for black box testing by non-developers.

If the QML module is mostly implemented in cpp, then QTestLib can be used for direct testing against the cpp layer. If the module is a plugin, this can lead to complications because Qt's build system does not support linking against a plugin. The module author can split their cpp code into a dynamic library that can be linked against and a plugin that only does QML type registration. This style of testing may be sufficient for some projects and we won't discuss it further.

Requirements for a comprehensive QML test solution

Running and managing tests

• Tests are written directly in QML/JavaScript, importing helpers from cpp where appropriate.
• Minimal cpp harness required in the user's test code, ideally no harness. The QML test framework should provide the harness.
• Running tests with qmlviewer for quick prototyping.
• Running tests automatically from a "make check" rule in a CI system.
• Reporting test results in QTestLib's standard test log formats (text, XML, XUnit) for CI reporting tools.

Basic Test API

• Equivalents of QCOMPARE, QVERIFY, QFAIL, etc.
• Data-driven tests.
• Benchmark support.
• Support for initTestCase/cleanupTestCase/init/cleanup.
• Report filename and line number for test failures.

Asynchronous testing

• Blocking: equivalents of QTRY_VERIFY, QTRY_COMPARE.
• Event based: "when" property that triggers a test.
• Testing states, transitions, and animations.
• Simulating keyboard and mouse events.

Reusing QTestLib

• Logging facilities: plain text, XML, XUnit for compatibility with the Qt build system and CI tools.
• Export some selected private API's for use by the QML plugin that provides the QtQuickTest namespace.
• Benchmark instrumentation support, particularly callgrind and CPU tick counters.
• Add exported functions to Qt 4.7 or 4.8?

Deployment

• As much as possible, deploy the same way as QML applications: copy directory to device and run qmlviewer or a QML test harness program.
• Could use qrc resources to bind the QML parts into a deployable binary, but it increases the maintainence burden on the test writer (deprecated - no longer a requirement).

Tooling

• Creator should provide a "Create QML unit test" option under File/New, and auto-completion of all test elements and helper functions.

System testing

QtUiTest uses a different style of testing from regular unit testing. It uses a "black box" approach where the tester is not aware of the internal structure of the application code. Instead of "check that the 'text' property on the 'surname' line edit widget is set to 'Smith'", it uses declarations of the form "verify that the contents of the line edit after the 'Surname:' label is 'Smith'". The specific widgets and mechanical layouts are ignored, with the focus on "what does the user see?". This is intended to give a "user view" of testing, suitable for simulating manual steps.

The cpp version QtUiTest relies upon the rich nature of Qt widget classes to make it easy to find "the QLineEdit associated with a specific QLabel". QML primitives make this more difficult - the basic building blocks of Rectangle, Text, Image, etc do not directly correspond to semantic notions on-screen. Instead, the application writer builds "widgets" as custom items out of primitives and then builds the application out of those custom items. The "text" or "value" property on the custom item is the value of interest to a system tester, not the primitive component properties.

What this means is that it isn't possible to provide a general system test framework for QML. However, it should be possible to provide system test support for specific QML component toolkits, such as Qt Components. JavaScript helper functions can be provided to assist the test writer; e.g. QtComponents.lineEditText("Surname:") could iterate over all custom line edit items to find the one that follows a custom label item with the text "Surname:", and then return the text property from the line edit item. This preserves the "what does the user see?" nature of system testing.

Another thing that QtUiTest does is to run an application in a completely separate process, proxying keyboard and mouse events, and property fetch operations. This is to preserve the idea that the application under test is not modified - it is used the way a user would. The problem with such instrumentation is that it limits the tester to the operations supported by the proxy protocol. QML provides a better method - the application under test can be loaded with Qt.createComponent() and then manipulated by the helper functions in-process within the test program. The full power of the QML test framework is available - unit test code and system test code can be mixed.

Visual Testing

The qmlvisual framework uses qmlviewer to record traces of application activity - screen shots, mouse events, etc. These traces can then be replayed later, or on different devices, to verify that the same results are produced.

import Qt.VisualTest 4.7

VisualTest {
    Frame {
        msec: 0
    }
    Frame {
        msec: 16
        image: "flickable-vertical.0.png"
    }
    Frame {
        msec: 32
        hash: "bca482a77823f03e8fb4170ee329fc0e"
    }
    ...
    Mouse {
        type: 2
        button: 1
        buttons: 1
        x: 159; y: 207
        modifiers: 0
        sendToViewport: true
    }
    Frame {
        msec: 304
        hash: "3d1b648229210ae5b57a0be51cc02f67"
    }
    ...
}

To reduce the disk space burden, some frames can be specified using their MD5 hash instead of a literal PNG image.

The Qt.VisualTest namespace is built into qmlviewer. It should probably be moved into a plug-in so that other test harnesses can make use of the visual test elements.

This form of "record and replay" visual testing is effective for checking animations frame-by-frame, but is difficult to use in conjunction with algorithmic testing.

Issues

• The QTestLib loggers (plain, xml, and xunitxml) assume that the name of the executable is the test case name, with only function names within that test case. For QML it can be better to use a single test running harness to execute several unrelated tests in the same binary. It would be useful if the QTestLib loggers could output a single log file with multiple unrelated test cases without faking function names (QTBUG-16419).
• Is it possible to get JavaScript code coverage information out of JSC/QtScript? Answer: yes, by code rewriting - see JSCoverage for an example. The QMLJS parser could be used for a similar purpose - it already supports rewriting.

Implementation

The following project is a more advanced version of Qt3D.Test, as a starting point for implementing the above requirements:

• http://qt.gitorious.org/qt-labs/qtest-qml

Test cases are written in QML as JavaScript functions within a TestCase element; for example:

import QtQuick 1.0
import QtQuickTest 1.0
Button {
    id: button
    onClicked: text = "Clicked"
    TestCase {
        name: "ClickTest"
        function test_click() {
            button.clicked();
            compare(button.text, "Clicked");
        }
    }
}

Results are reported via QTestLib using the standard test loggers:

********* Start testing of ClickTest *********
Config: Using QTest library 4.7.2, Qt 4.7.2
PASS   : ClickTest::initTestCase()
PASS   : ClickTest::test_click()
PASS   : ClickTest::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of ClickTest *********

Any function whose name starts with "test_" is run as a unit test. Functions whose names start with "benchmark_" are run as benchmarks within an implicit QBENCHMARK context:

import QtQuick 1.0
import QtQuickTest 1.0
TestCase {
    id: top
    name: "CreateBenchmark"
    function benchmark_create_component() {
        var component = Qt.createComponent("item.qml")
        var obj = component.createObject(top)
        obj.destroy()
        component.destroy()
    }
}

********* Start testing of CreateBenchmark *********
Config: Using QTest library 4.7.2, Qt 4.7.2
PASS   : CreateBenchmark::initTestCase()
RESULT : CreateBenchmark::benchmark_create_component:
     0.22 msecs per iteration (total: 57, iterations: 256)
PASS   : CreateBenchmark::benchmark_create_component()
PASS   : CreateBenchmark::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of CreateBenchmark *********

Modifications needed to QTestLib

Option 1 - Export needed functions, no major library additions:

• Export QTestResult, QTestLog, and qParseArgs()
• Add a "qml" flag to qParseArgs() to cause it to store function names for running specific tests. Export the storage locations.
• Benefit: Gives maximum flexibility to QtQuickTest infrastructure to change.
• Drawback: Changes the binary compatibility on QTestLib.

Option 2 - New private, but exported, API

• Add QuickTestResult class to QTestLib - a QObject-based class with no QtDeclarative dependencies, that can be easily QML-ized.
• Benefit: Existing QTestLib classes retain their current export status. Binary change is only by addition.
• Drawback: Need to get it right first time to minimize disruption from further changes.