Details
-
Task
-
Resolution: Done
-
P1: Critical
-
None
-
None
Description
All autotests should be running in Qt5 CI.
I would like to do this a bit differently than for Qt4.
The Qt4 way:
- compile, run all autotests via (private) scripts
- get results in testlib XML files
- upload results to (private) database
- download results from (private) database frontend
- subtract results for tests marked as unstable in a (private) blacklist
- compare munged results against last results from (private) database, consider test run "successful" iff no new regressions appeared
The Qt5 way should be:
- compile, run all autotests, via scripts in public qtqa module
- leveraging build system as much as possible (i.e. `make -j1 -k check')
- test output goes to two places: stdout as plain text, and to XML log files
- any test exiting with a non-zero exit code means the test run fails
- unstable tests are handled by marking them as unstable directly in the source (e.g. by extending QEXPECT_FAIL)
- unstable tests which may crash may need something more, e.g. CONFIG+=insignificant_test in .pro file
- XML log files may be uploaded to (private and/or public) results handling systems
- only for gathering of metrics - test results database no longer handles any regression detection logic
Probably the best way to get this going is to pick a small module (qtsvg ?) where all the tests are stable and passing, and implement the basic test running for that module first, then gradually extend to other modules.
Attachments
There are no Sub-Tasks for this issue.