summaryrefslogtreecommitdiffstats
path: root/src/testlib/qtestresult.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Doc: Document QTouchEventWidgetSequence, QTestResultTopi Reinio2025-08-271-0/+6
| | | | | | | | | | | | | | | | | | | | | In Qt 6, widget-specific functions for creating touch event sequences were moved to a new class, QTouchEventWidgetSequence. We mention this class in our porting guide and document its member functions, but the class itself was not documented. Adding the \class topic clears QDoc warnings such as: (qdoc) warning: No output generated for function 'QTest::QTouchEventWidgetSequence::press()' because 'QTouchEventWidgetSequence' is undocumented Similar warnings are generated for class QTestResult with a private implementation; add \class topic for it but mark it \internal. Pick-to: 6.10 6.9 6.8 Change-Id: I642dc4b160601247a6938b228caff788f655c866 Reviewed-by: David Boddie <david.boddie@qt.io> Reviewed-by: Paul Wicking <paul.wicking@qt.io>
* Replace qTerminate() with std::terminate() and mark it for removalThiago Macieira2024-12-261-1/+1
| | | | | | | | | | | | It was used by the QT_TERMINATE_ON_EXCEPTION macro, introduced in 2012, to support pre-C++11 noexcept semantics. That macro was removed for Qt 6.8 in commit 9b2ae564a59656d9cf49b141e70f5958b4fb79a4. This commit amends that removing the definition of qTerminate() immediately in Qt 6.9 (it was an \internal function). Pick-to: 6.9 Change-Id: I9682121c04fafb3676b0fffd9f5ac999e7603c84 Reviewed-by: Ahmad Samir <a.samirh78@gmail.com>
* Mark max length work buffers uninitializedAllan Sandfeld Jensen2024-11-301-4/+4
| | | | | | | | | Otherwise the hardening with initializing all buffers causes serious performance regressions Pick-to: 6.8 Change-Id: I3f7a0b7f0e0d08644b1dbb520cf1f6d5e052b270 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* [QCOMPARE_3WAY] Expected and actual vars should use different formattersTatiana Borisova2024-11-211-3/+4
| | | | | | | | | | | - Split the orderFormatter into two parameters: actualOrderFormatter and expectedOrderFormatter Ammends 454f010e58b4f17df5d2683233e92448097d2dd1 Change-Id: I4ec468a0c0a4daea41c5ba2ce5ccfff0378460b8 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* Create a QCOMPARE_3WAY macro to test the C++20 spaceship operator <=>Tatiana Borisova2024-11-141-2/+89
| | | | | | | | | | | | | | | | | Since the C++20 spaceship operator <=> appeared in the standard, qtest framework starts using it for tests. To make the way of usage the spaceship operator convenient and informative, the QCOMPARE_3WAY macro is added. - Add QCOMPARE_3WAY macro usage testcases. - Add threewaycompare test to setftests. [ChangeLog][QtTest] Added the QCOMPARE_3WAY macro. The macro tests the C++20 spaceship operator <=> Fixes: QTBUG-104108 Change-Id: Ia14b26c1d70625ac8c6cf2278d597b2a8cbe31d0 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Rename qt_saturate to q26::saturate_castGiuseppe D'Angelo2024-08-181-3/+2
| | | | | | | | | | | | C++26 adds std::saturate_cast, so follow the established pattern of other similar "backported" APIs. The old name is left around while we port other submodules. While at it, move qt_saturate's tests to the qnumeric test. Pick-to: 6.8 Change-Id: I653a2e3d936081378298a9c8e51e7c1a2d438d83 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QtTest: mark more functions as coldMarc Mutz2024-08-031-0/+2
| | | | | | | | | | | | Unit test failure should not happen, so mark failure formatters as cold (like qFail() itself). As a drive-by, mark formatFailMessage() overloads file-static, too. Pick-to: 6.8 Change-Id: I92f5b674a7282f4ebaa9250894a9971a524f4108 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QtTest: port from qsnprintf to std::snprintf and mark the module as ↵Marc Mutz2024-08-031-22/+24
| | | | | | | | | | | | qsnprintf-free Drive-by remove an explicit NUL-termination (std::snprintf() does that) and port a repeated use of printf argument checking to the protect() idiom. Pick-to: 6.8 Change-Id: Ida15940fe9aef0622e9836a229a398c909503a9a Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* QPROPERTY_TEST_COMPARISON_HELPER: fix extra leading \n in failure outputMarc Mutz2024-08-011-1/+1
| | | | | | | | | | | | | Once we ported the formatting to use formatFailMessage() in aa8a4be9e9eae82b6c6b8875371114dbfb3ec06a, it became apparent that the output contained an extra leading newline compared QCOMPARE use of formatFailMessage(). We kept the newline in the patch that was picked to older branches. Since this is a stylistic discrepancy, though, standardize on no leading newline going forward. Pick-to: 6.8 Change-Id: I4badf700900a106d835f3f72e92441d8a8912ff7 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QtTest: port approx_wide_len() from std::clamp() to qt_saturate()Marc Mutz2024-07-311-4/+4
| | | | | | | | | | The patch that originally added the function used std::clamp() because qt_saturate() wasn't available in all the branches to which the patch was backported. This patch modernizes the code for newer branches. Pick-to: 6.8 6.7 6.5 Change-Id: I1b764d303e00ec04858643efed3dcc71f2c7ce4c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QPROPERTY_TEST_COMPARISON_HELPER: fix mbstowcs and snprintf usageMarc Mutz2024-07-311-0/+11
| | | | | | | | | | | | | | | | | The mbstowcs() function is not re-entrant, and while we can probably rule out that our test macros are being executed concurrently, we can't rule out that other parts of the test concurrently execute std::mbstowcs(). To fix, de-inline the code so we can re-use the existing formatFailMessage() in qtestresult.cpp, which has already been fixed before. Amends 930e59b798d9e3d08e17440980d33a08fb411cbe. Pick-to: 6.8 6.7 6.5 6.2 Change-Id: I61144af13a41ea0b4fba17bd232e660ef33dbd20 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest::formatFailMessage(): fix use of std::mbstowcs()Marc Mutz2024-07-311-2/+18
| | | | | | | | | | | | | | | | | | | | | | | | | This function is not re-entrant, and while we can probably rule out that our test macros are being executed concurrently, we can't rule out that other parts of the test concurrently execute std::mbstowcs(). Use the re-entrant version and also fix the format string (%* requires int, not size_t). Saturation is ok here, because, while std::mbsrtowcs() will happily return a value > maxMsgLen when dst == nullptr and the string is longer, we will never actually print more than maxMsgLen of it. As a drive-by, port to std and adjust the QNX comments to describe the quirky QNX behavior, if, indeed, still present in the <cwchar> version, better. This author had to go back in the git history to figure out what it wanted to tell him. Amends 2d8028d696a86102a7753f9d59fb41f4170181a8 (mbstowcs) and d040681b6f3d03b349e9b9487fe89a611d03ee3c (%*). Pick-to: 6.8 6.7 6.5 6.2 5.15 Change-Id: I6215713c643647727f73f4e2f2a7ac34204af40d Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: rip out qxp::function_ref from reportResult()Thiago Macieira2024-04-261-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is causing huge code bloat because everything is a local lambda. Instead, pass direct type-erased function and data pointers to the replacement function. Testing with tst_qcborvalue, this reduces the compilation time and the output binary size significantly: Before After Compiler Time Size Time Size GCC 13.2 136.99 s 202.3 MB 13.88 s 14.3 MB GCC 14.0 131.49 s 202.7 MB 14.69 s 14.4 MB Clang 17 77.2 s 146.7 MB 13.62 s 12.2 MB Clang 18 141.9 s 187.1 MB 13.62 s 12.4 MB This causes a difference in how toString() overloads are found. Previously it would match far more overloads because the toString() calls were expanded by the macro. Now, we depend on Argument-Dependent Lookup and associated namespaces, so toString() overloads should not be in the QTest namespace any more. With this patch applied, the testlib testcase of tst_selftest started failing, because nullptr is now handled differently. However, I consider it as a bugfix, because previously it was falling back to a default implementation, and now it is using the QTest::toString(std::nullptr_t) overload, which is a desired behavior. Update the reference files for tst_selftest with the new expected output. Task-number: QTBUG-124272 Change-Id: Ie28eadac333c4bcd8c08fffd17c5484186accdf6 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: make failOnWarning() functional on temp objects destructionVladimir Belyavsky2023-11-221-7/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | We need to be able to handle warnings that may occur when temporary objects, that were created in a test function, are destroyed. For example, now we miss all warnings that might be triggered from the object destructor, if the object's deletion was delayed (e.g. via deleteLater()). Also we miss all the warnings that were triggered on the test's cleanup() call. To fix this we need simply move QTestLog::clearFailOnWarnings() from QTestResult::finishedCurrentTestData() to the later stage, i.e. into QTestLog::clearCurrentTestState() which is actually called in appropriate time from QTestResult::finishedCurrentTestDataCleanup(). Same for QTestLog::clearIgnoreMessages(), since they are interrelated, so we need to clear them at the same time. We need this change for QML tests in particularly, to be able fail on warnings that might be triggered from Component.onDestruction() of some temporary test object. Pick-to: 6.6 6.5 Change-Id: I58a57691f20761619f56bd1bea3a862f2c26f569 Reviewed-by: Mitch Curtis <mitch.curtis@qt.io> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Increase QTest failure message limitMitch Curtis2023-10-191-5/+9
| | | | | | | | | | | | | | QPalette specifically has quite a large amount of output (1363 characters) when toString is called on it. We should make sure that we can fit that in our failure messages. This patch does that by increasing the limit from 1024 characters to 4096. Fixes: QTBUG-5903 Fixes: QTBUG-87039 Pick-to: 6.5 6.6 Change-Id: I1dc5078ad05858bb6542c3a06c6b84711af79e4f Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* QTestEventLoop: don't stop the test fails outside the main threadThiago Macieira2023-05-051-1/+2
| | | | | | | | | | | | | | | | | | | | | This amends commit 115f828ae406d2805869fccda9269209af646a84 "QTestEventLoop: stop when the test fails" from Qt 6.3 to not attempt to stop the QTestEventLoop instance if the failure happened outside the main thread. This is to prevent the QTEL from being created outside the main thread and thus cause warnings about QObject in addition to and logged before the real test failure. [ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop no longer attempts to exit its event loop if the failure was detected outside the main thread. QWARN : tst_QMutex::tryLock_non_recursive() QObject: Cannot create children for a parent that is in a different thread. (Parent is QCoreApplication(0x7fffffffe710), parent's thread is QThread(0x802e29000), current thread is QThread(0x7fffffffd5a8) FAIL! : tst_QMutex::tryLock_non_recursive() 'timer.elapsed() >= waitTime - systemTimersResolution' returned FALSE. () Change-Id: Ieab617d69f3b4b54ab30fffd175b139322a1f02c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTest: Rename QCOMPARE_XX parameters to add meaningMårten Nordheim2023-03-081-8/+8
| | | | | | | | | | | | | | | The parameter names were previously "lhs" and "rhs", which is incredibly abstract for a testing framework. One of the parameters will tend to be a baseline value to compare against while the other is the result of some action we want to test. Thus I suggest they be renamed "computed" and "baseline". This way we can, hopefully, retain the semantic that the 'left'/first argument is the computed ('actual' in QCOMPARE) value while the 'right'/second argument is the baseline ('expected' in QCOMPARE.) Change-Id: I3e0fdce2a3f1faca06fdf7184ef6e0eb9724d990 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* Long live Q_UNREACHABLE_RETURN()!Marc Mutz2022-10-151-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combination of Q_UNREACHABLE() with a return statement. ATM, the return statement is unconditionally included. If we notice that some compilers warn about return after __builtin_unreachable(), then we can map Q_UNREACHABLE_RETURN(...) to Q_UNREACHABLE() without having to touch all the code that uses explicit Q_UNREACHABLE() + return. The fact that Boost has BOOST_UNREACHABLE_RETURN() indicates that there are compilers that complain about a lack of return after Q_UNREACHABLE (we know that MSVC, ICC, and GHS are among them), as well as compilers that complained about a return being present (Coverity). Take this opportunity to properly adapt to Coverity, by leaving out the return statement on this compiler. Apply the macro around the code base, using a clang-tidy transformer rule: const std::string unr = "unr", val = "val", ret = "ret"; auto makeUnreachableReturn = cat("Q_UNREACHABLE_RETURN(", ifBound(val, cat(node(val)), cat("")), ")"); auto ignoringSwitchCases = [](auto stmt) { return anyOf(stmt, switchCase(subStmt(stmt))); }; makeRule( stmt(ignoringSwitchCases(stmt(isExpandedFromMacro("Q_UNREACHABLE")).bind(unr)), nextStmt(returnStmt(optionally(hasReturnValue(expr().bind(val)))).bind(ret))), {changeTo(node(unr), cat(makeUnreachableReturn, ";")), // TODO: why is the ; lost w/o this? changeTo(node(ret), cat(""))}, cat("use ", makeUnreachableReturn)) ); where nextStmt() is copied from some upstream clang-tidy check's private implementation and subStmt() is a private matcher that gives access to SwitchCase's SubStmt. A.k.a. qt-use-unreachable-return. There were some false positives, suppressed them with NOLINTNEXTLINE. They're not really false positiives, it's just that Clang sees the world in one way and if conditonal compilation (#if) differs for other compilers, Clang doesn't know better. This is an artifact of matching two consecutive statements. I haven't figured out how to remove the empty line left by the deletion of the return statement, if it, indeed, was on a separate line, so post-processed the patch to remove all the lines matching ^\+ *$ from the diff: git commit -am meep git reset --hard HEAD^ git diff HEAD..HEAD@{1} | sed '/^\+ *$/d' | recountdiff - | patch -p1 [ChangeLog][QtCore][QtAssert] Added Q_UNREACHABLE_RETURN() macro. Change-Id: I9782939f16091c964f25b7826e1c0dbd13a71305 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Don't print QCOMPARE values if they lack string representationVolker Hilsheimer2022-07-101-23/+27
| | | | | | | | | | | | | | | | | | | | | | | Before 0681a2dd5a8095baddb5905fb21a58ce19b958c5, QCOMPARE'ing types for which no QTest::toString specialization exists did not output Actual and Expected lines on failure, as that would only print <null> for both values (which then look like the same value, confusingly). Commit 0681a2dd5a8095baddb5905fb21a58ce19b958c5 changed that behavior, and started printing the confusing <null> values. Take care of the logic in the formatFailMessage function: if both values are nullptr, then print only the variable names, but not the confusing <null> text representation of the values. Remove dead and duplicated code related to the formatting logic, add a self-test function, and update the expected_cmptest files. Fixes: QTBUG-104867 Pick-to: 6.4 Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QCOMPARE/QVERIFY: fix huge pessimisation in QTestResult::reportResultMarc Mutz2022-06-031-1/+2
| | | | | | | | | | | | | | | | The old code allocated a stack buffer, but asked the runtime to zero-initialize it. That's 1KiB of writes to the stack on every QCOMPARE and QVERIFY before any actual work is done. Fixing this little laissez-faire to just initialize the first character in the buffer results in nice little speedups of ~40%. This fixes the issue in reportResult(), a Cut'n'paste from compare() and verify(), which have since been fixed by a previous commit. Change-Id: I5cad57299490925b88e768dc751304699274db2e Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestLib: rework QTest::compare_helper()Ivan Solovev2022-06-031-2/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its arguments lazily, speeding up the general case where the comparison doesn't fail. This is true for the QCOMPARE functionality provided by Qt. If you specialized qCompare() for your own types, then you need to change its implementation in line with Qt's own qCompare() specializations in order to enable this feature. [ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will now print the actual and expected values upon failure. Previously it was not like that because of the compareHelper() overload in qtestresult.cpp that treated the presence of nullptr-arguments as a reason to ignore formatFailMessage() call. New implementation does not have this check, and correctly executes formatFailMessage() for all arguments. Note that the qCompare() overloads that call QTestResult::compare() internally were not affected by this patch, because they already defer toString() invocation until the comparison fails. Some numbers, collected against shared release developer build. I checked how this change affects the test execution. The idea was to pick some tests for types that do not have a specific QTestResult::compare overload, so I picked a couple of QByteArray tests. The comparison is done by running a test 10 times and taking the average execution duration, as reported in the log. tst_qbytearrayapisymmetry: Before: 15.6 ms After: 14.2 ms tst_qbytearray: Before: 41 ms After: 36 ms The benefit is around 9% and 12% respectively. Fixes: QTBUG-98874 Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()Ivan Solovev2022-06-031-11/+105
| | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}() macros. These new macros behave similarly to QVERIFY(a op b), where 'op' is ==, !=, <, <=, >, >= respectively, but print a formatted error message with argument values in case of failure. The formatting is done lazily, which means that the strings will be generated only when the comparison fails. Also add a new test for tst_selftest and generate expected output for it. Fixes: QTBUG-98873 Task-number: QTBUG-98874 Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QCOMPARE/QVERIFY: fix huge pessimisation in QTestResultMarc Mutz2022-06-031-2/+4
| | | | | | | | | | | | | | | | | | | | | | | The old code allocated a stack buffer, but asked the runtime to zero-initialize it. That's 1KiB of writes to the stack on every QCOMPARE and QVERIFY before any actual work is done. Fixing this little laissez-faire to just initialize the first character in the buffer results in nice little speedups of up to or exceeding 2x. Amends d946507727b363326d05f48da93c2af04bdda76d. [ChangeLog][QtTestLib] Optimized successful QCOMPARE and QVERIFY for an up to 2x speedup. This has the potential to meaningfully reduce the load on the CI, so picking all the way to 5.15. Pick-to: 6.3 6.2 5.15 Change-Id: Ib93d69282ec87cbd26a60e4ac14413e8cef8ff78 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-161-38/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* QtTestLib: replace remaining uses of QL1String with QL1StringViewSona Kurazyan2022-05-021-2/+2
| | | | | | | | | Remove unneeded \fn qdoc lines as a drive-by. Task-number: QTBUG-98434 Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Make counts of various types of test result add up correctlyEdward Welbourne2022-01-131-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Add QTest::failOnWarningMitch Curtis2021-11-261-0/+1
| | | | | | | | | | | | | | This solves the long-standing problem of not being able to easily fail a test when a certain warning is output. [ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test function, any warning that matches the given pattern will cause a test failure. The test will continue execution when a failure is added. All patterns are cleared at the end of each test function. Fixes: QTBUG-70029 Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Let loggers know which test function is being leftTor Arne Vestbø2021-08-101-2/+2
| | | | | | | | | | | None of the loggers rely on this at the moment, but one could imagine one that does. Plus, doing any sort of debug logging in leaveTestFunction will at the moment be attributed to UnknownTestFunc(). Pick-to: 6.2 Change-Id: I284b2785a276e028b9f57c26357679fd9e045ca7 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Document order of QTestResult completion functionsTor Arne Vestbø2021-08-101-0/+39
| | | | | | | | Pick-to: 6.2 Change-Id: I34adbcc2bd02887a93cd1e86692401e840afe8bd Reviewed-by: Paul Wicking <paul.wicking@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestEventLoop: stop when the test failsEdward Welbourne2021-08-051-0/+2
| | | | | | | | | | | | | | | | | | | | | | It makes no sense for the event loop of a test to keep running after a test has failed. This lets test code simply use the usual testlib macros to compare and verify values in asynchronous tests that would otherwise need to hand-test values and send a signal on failure (that the main test can connect to an event-loops quit() or equivalent). For example, QLocalSocket's benchmark simply uses the usual macros, without doing anything to stop its event loop if they fail, with the sad result that, when a test fails, it does so repeatedly and then times out, causing the test program to be killed without running later tests. With this change, that test code (once converted to use QTestEventLoop) is able to exit gracefully on the first failure. [ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop new exits its event loop as soon as the test is known to be failing. Task-number: QTBUG-91713 Change-Id: If0d455741668722034906763025dda496d2afbb4 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Pass on file location on failure, but don't assume we have oneTor Arne Vestbø2021-08-041-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We try our best to pass on the file location of a failure, including for fatal errors, but the reporting or logging machinery should not assume there is one. By passing on nullptr for the file location we allow the logging backends to decide how to handle the situation, e.g. by not emitting extra fields for failure location. This effectively reverts c25687fa0b6e4be043e1f8c92c093d8b06ca06c4, in favor of relying on the backends to cope with null filename, which they already did. As qFatal uses QMessageLogger, which by default disables file/line information in release builds, we need to explicitly enable this in our self-tests, to get uniform test results. Similarly, we disable file/line info from testlib itself, as reporting Qt internal file and line information for user diagnostics is less useful. The odd one out there is qtestdata.cpp, which still ends up in test output due to using QTEST_ASSERT instead of qFatal for its diagnostics. Cleaning up that, and unifying how we report testlib issues to the user, is left for another day. Pick-to: 6.2 Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Fix QFAIL() to interract correctly with QEXPECT_FAIL()Edward Welbourne2021-06-151-6/+11
| | | | | | | | | | | | | | | | | | | Previously, it went direct to QTestResults::addFailure() without going via the checking for expected failure. Add QTestResults::fail() to take care of this checking, as for verify() and compare(). Tidied up the code implementing expected failure and QFAIL(), while I was about it. Adjusted an existing test to verify that expecting a QFAIL() works, by using QFAIL() instead of QVERIFY(false). Remove the QVERIFY(false) whose comment brought this to my attention. [ChangeLog][QtTestLib][QFAIL] QEXPECT_FAIL() now correctly anticipates a subsequent QFAIL(). Previously QFAIL() counted as a fail regardless. Change-Id: Icc28cf70e5ff3006363791ea03aa01f2f591eb71 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestlib: Speed up comparison of qsizetypeFriedemann Kleint2021-01-221-0/+10
| | | | | | | | | | | | | Add a qCompare() overload for qsizetype, preventing it falling back to the slow code path pre-formatting expected/actual. toString() should use the correct format from the already present int-types (see macro TO_STRING_IMPL). Complements 94aa350621e8a5c4ad3b438c10fc1c0a9ed3bc8a. Task-number: QTBUG-38890 Change-Id: Ieb8cea7de086141a2c80f93b4c1be01572be96df Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTestLib: add a abort-on-fail environment variableGiuseppe D'Angelo2020-12-041-11/+41
| | | | | | | | | | | | | | | | | | | | | When debugging a spurious failure it's extremely useful to run the test repeadtly into a debugger until a failure appears. When that happens, one wants to immediately start debugging. In so far, this has only been possible by placing breakpoints inside Qt itself (when a failure is logged). Add another way: an env variable, similar to QT_FATAL_WARNINGS, that makes any failure fatal (terminate() gets called. Bonus: you can control the termination handler!) [ChangeLog][QtTestLib][QtTest] When the QTEST_FATAL_FAIL environment variable is set to a non-zero value, a test immediately aborts its execution. This is useful to debug intermittent failures. Change-Id: If2395f964ea482c30b8c8feab98db7fdee701cd3 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Jason McDonald <macadder1@gmail.com>
* QTestResult: Don't pass a nullptr file to addFailure()Ulf Hermann2020-09-301-3/+5
| | | | | | | | addBFail() asserts on the file being non-null. The convention seems to be "Unknown File" for cases where we cannot determine the file. Change-Id: I3a4d0130352d77d75f264fad6f3bd47c6700ef4c Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
* QTestLib: Speed up QCOMPARE for QStringFriedemann Kleint2019-08-011-0/+26
| | | | | | | | | | | | Add overloads for qCompare() for QStringView making use of the fast formatting helper introduced by 94aa350621e8a5c4ad3b438c10fc1c0a9ed3bc8a for int. Speeds up the bug report example by a factor of 3..4. Task-number: QTBUG-38890 Change-Id: Icc706618b2f1d23b37d354a04d4e1d1cc4b5aee3 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTestLib: Introduce nullptrFriedemann Kleint2019-07-111-19/+17
| | | | | | | | Apply Fixits by Qt Creator with some amendments. Task-number: QTBUG-69413 Change-Id: I620e40a277dc2b20b0ec26fc32577e66b0456bb3 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* QTestLib: Speed up QCOMPARE for float, double, int, unsignedFriedemann Kleint2019-07-091-15/+101
| | | | | | | | | | | | Factor out a helper template formatting the QCOMPARE failure message delaying the formatting of the parameters with toString() and use that for float, double, int and unsigned. This removes the need to always format and allocate strings for the operands even in the success case, speeding up the QColor test from 3.3s to 700ms (Windows/release). Task-number: QTBUG-38890 Change-Id: I999484765bdaed921d3fc35f35a9fbbcd82a9704 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Add BXPASS and BXFAILOliver Wolff2019-01-291-3/+10
| | | | | | | | | | | | | | | Prioritize blacklisting over QEXPECT_FAIL so that a test that is blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To reflect this state properly, the two values of BXPASS and BXFAIL were added to testlib's output. [ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests will be taken into account for XPASS and XFAIL. A blacklisted test that causes an XPASS will no longer be a fail. Task-number: QTBUG-72928 Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Add Test Anything Protocol (TAP) reporterTor Arne Vestbø2018-03-141-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Test Anything Protocol (TAP), was originally Perl's simple text-based interface between testing modules and test harnesses, but has since been adopted by a large number of producers and consumers in many different languages, which allows colorizing and summarizing test results. The format is very simple: TAP version 13 ok 1 - test description not ok 2 - test description --- message: 'Failure message' severity: fail expected: 123 actual: 456 ... ok 3 - test description # SKIP 1..3 The specification [1] is very brief, so the implementation has been based on how typical consumers behave, especially when it comes to the undefined diagnostics block. [1] http://testanything.org/tap-version-13-specification.html Change-Id: I616e802ea380165c678510e940ddc6607d39c92d Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
* Updated license headersJani Heikkinen2016-01-151-14/+20
| | | | | | | | | | | From Qt 5.7 -> LGPL v2.1 isn't an option anymore, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new LGPL header instead of LGPL21 one (in those files which will be under LGPL v3) Change-Id: I046ec3e47b1876cd7b4b0353a576b352e3a946d9 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
* Testlib: Avoid formatting unneeded messages.Friedemann Kleint2015-10-301-9/+11
| | | | | | | | | Brings down gui/painting/qcolor from 7s to 3s, reducing the calls to printf() helpers from 16E6 to 10E6. Task-number: QTBUG-38890 Change-Id: I34065e6f09fc9a14920b06aa6936908229f3f9c4 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
* Update copyright headersJani Heikkinen2015-02-111-7/+7
| | | | | | | | | | | | | | | | | | Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
* Update license headers and add new license filesMatti Paaso2014-09-241-19/+11
| | | | | | | | | - Renamed LICENSE.LGPL to LICENSE.LGPLv21 - Added LICENSE.LGPLv3 - Removed LICENSE.GPL Change-Id: Iec3406e3eb3f133be549092015cefe33d259a3f2 Reviewed-by: Iikka Eklund <iikka.eklund@digia.com>
* Add support for blacklisting test functionsLars Knoll2014-09-211-2/+15
| | | | | | | | | | | | | | | | | We need to have a finer grained control over the tests we skip in our CI system. This adds a blacklisting mechanism that allows blacklisting individual test functions (or even test data) using a set of predefined matching keys for the operating system and some other relevant variables. QTestlib will search for a file called BLACKLIST in the test directory and parse it if found. The file contains a simple ini style list of functions to blacklist. For details see qtestblacklist.cpp. Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
* Fix auto test failure output on QNXBernd Weimer2014-05-191-8/+9
| | | | | | | | | On QNX mbstowcs does not ignore last parameter (max) when first (dest) is NULL. Set it to sufficiently large value to yield proper results on QNX. Other platforms (standard libraries) will ignore this value anyway. Change-Id: Ie4695254d45082e151a052bf16de684af3b1ba1e Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Remove duplicate symbol name: QTest::currentAppNameThiago Macieira2014-04-301-4/+4
| | | | | | | | | | It's declared in qtestcase.h as a function, so let's not declare the one in qtestresult.cpp as a static variable. None of the variables in qtestresult.cpp need to be in the QTest namespace, but we don't need to change them now. Change-Id: If6cc34642fdfe3ccda3b8cea7d053ead0db9ccbd Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
* Fix the alignment for non-ASCII stringsThiago Macieira2014-01-221-2/+4
| | | | | | | | | | | | | | | | | | d040681b6f3d03b349e9b9487fe89a611d03ee3c added support for aligning the test results for easier side-by-side comparison of the actual and expected values. However, it didn't take into account multibyte strings. That is, we would see: FAIL! tst_testcase::testcase: Compared values are not the same Actual (QString("é")): F0O Expected (expected) : FOO We use mbstowcs (multibyte string to wide char string) that calculates the length in wide chars of the output string. That's roughly equivalent to QString::fromLocal8Bit(string).toUcs4().size(). Change-Id: Ic2649951c50e05143da32a7fbef00a01e385c542 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Expose QTest::currentAppName() and remove hard-coded argv[0] in testsTor Arne Vestbø2013-09-261-5/+5
| | | | | | | | Except where we're actually testing QCoreApplication::applicationName() and friends. Change-Id: I25514884c11f43a4f82b1f818f822dc3d79f69a3 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@digia.com>
* TestLib: align the colons so the "actual" and "expected" align tooThiago Macieira2013-07-211-3/+5
| | | | | | | | | | | | | | | Before: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual): F0O Expected (expected): FOO Now: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual) : F0O Expected (expected): FOO Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef Reviewed-by: Jason McDonald <macadder1@gmail.com>
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-16 10:43:33 +0000