summaryrefslogtreecommitdiff
path: root/run_project_tests.py
AgeCommit message (Collapse)Author
2020-06-13tests: reset CMakeDependency.class_cmakeinfoDaniel Mensinger
2020-05-26Merge pull request #7197 from jon-turney/test-output-check-mandatoryJussi Pakkanen
Make the expected output check mandatory for failing-meson and warning-meson tests
2020-05-23Clear internal caches before running each test.Jussi Pakkanen
2020-05-23Make expected stdout mandatory for warning-meson and failing-meson testsJon Turney
Unify present or absent test.json file cases in gather_tests Make expected stdout mandatory in test.json for some test categories Use a trivial TestCategory class rather than a tuple, to make it easier to default category attributes
2020-05-23Remove stray print call.Jussi Pakkanen
2020-05-23Revert "Merge pull request #7172 from jon-turney/test-output-check-mandatory"Jussi Pakkanen
This reverts commit 0871b1032c53287a1ed3ce5108799fb0daccaec5, reversing changes made to 9dc3ca2c1c9fbb47e731551c6432df144f725261.
2020-05-22Merge pull request #7172 from jon-turney/test-output-check-mandatoryJussi Pakkanen
Make the expected output check mandatory for failing-meson and warning-meson tests
2020-05-15Make expected stdout mandatory for warning-meson and failing-meson testsJon Turney
Unify present or absent test.json file cases in gather_tests Make expected stdout mandatory in test.json for some test categories Use a trivial TestCategory class rather than a tuple, to make it easier to default category attributes
2020-05-14interpreterbase: Allow passing an extra message in feature/deprecation warningsDylan Baker
The intended use it to tell people the new thing to do.
2020-05-13Merge pull request #6620 from jon-turney/test-output-checkDylan Baker
Add a mechanism for validating meson output in tests
2020-05-08rename unstable-kconfig to unstable-keyvalPaolo Bonzini
Discussions in #6524 have shown that there are various possible uses of the kconfig module and even disagreements in the exact file format between Python-based kconfiglib and the tools in Linux. Instead of trying to reconcile them, just rename the module to something less suggestive and leave any policy to meson.build files. In the future it may be possible to add some kind of parsing through keyword arguments such as bool_true, quoted_strings, etc. and possibly creation of key-value lists too. For now, configuration_data objects provide an easy way to access quoted strings. Note that Kconfig stores false as "absent" so it was already necessary to write "x.has_key('abc')" rather than the more compact "x['abc']". Therefore, having to use configuration_data does not make things much more verbose.
2020-04-30Add a mechanism for validating meson output in testsJon Turney
Expected stdout lines must match lines from the actual stdout, in the same order. Lines with match type 're' are regex matched. v2: Ignore comment lines in expected_stdout v3: Automatically adjust path separators for location in expected output v4: Put expected stdout in test.json, rather than a separate file
2020-04-30Make colourize_console() a functionJon Turney
Currently, colourize_console is a constant, set at process initialization. To allow the actual stdout to be easily compared with the expected when running tests, we want to allow colourization to be on for the test driver, but not for the in-process configure done by run_configure, which has stdout redirected from a tty to a pipe. v2: Cache _colorize_console per file object v3: Reset cache on setup_console()
2020-04-28tests: Skip hotdoc test if hotdoc is not installedDaniel Mensinger
2020-04-28tests: Add support for specifying tool requirementsDaniel Mensinger
Adds the `tools` section to `tests.json` to specify requirements for the tools in the environment. All tests that fail at least one tool requirements check are skipped.
2020-04-17run_project_tests: Allow matrix tests to skipDylan Baker
Since they generate some tests that are only valid on specific operating systems.
2020-04-17run_project_tests: Add an option to put temprorary files in /tmpDylan Baker
I have an NVME drive, I really don't want to be thrashing it with temporary files. It's also annoying to watch vscode thrash about with files that are quickly created and deleted. I'd rather put them in /tmp, so I've added a non-default option to do so.
2020-03-20Merge pull request #6636 from jon-turney/machine-detection-problemsJussi Pakkanen
Redetect machines when languages change
2020-03-09project_tests: Add the option to the test format to mark the languageDylan Baker
This is needed when mixing D and C code, as it's possible to end up witha combination of linkers and compilres such that C produces pdb files but D does not.
2020-03-09run_project_tests: Sort importsDylan Baker
It was impossible to figure out what was coming from where before.
2020-03-09run_project_tests: fix argument typeDylan Baker
2020-03-09tests: Add pdb files for d testsDylan Baker
2020-03-09project_tests: Add "version" to "shared_lib" and "pdb" typesDylan Baker
This allows the harness to apply the version correctly, putting it in the right place, dropping the right amount of numbers, etc. pdb taking a version allows it to be more easily copied from the shared_lib type.
2020-03-06project_tests: Add a "shared_lib" typeDylan Baker
This allows fixing tests that produce .dylib's on macOS and .so's on elf Unices.
2020-02-29Extend _run_test to test build without regenerateJon Turney
Add coverage of a simple 'configure then build' sequence, to catch bugs which occur with that. Also, to simplify returning results from functions called by _run_test(), allow it also throw TestResult objects, which run_test() catches and returns.
2020-02-26test: replace pass_* functions with a test.json entryDaniel Mensinger
2020-02-25test: merge installed_files.txt into test.jsonDaniel Mensinger
2020-02-25test: realise test_args.txt with a test matrixDaniel Mensinger
2020-02-25test: merge test_matrix.json and setup_env.json into test.jsonDaniel Mensinger
2020-02-25test: general refactoring of run_project_tests.pyDaniel Mensinger
2020-02-20boost: Updated testsDaniel Mensinger
2020-02-13Remove unused no-installed-files test featureJon Turney
Looking at 45c8557d, the idea behind this seems to be that a test could conditionally indicate that the list of installed files should not be validated by creating that file. It's no longer used anywhere. Also remove a lingering no-install-files file which isn't used since commit c693bd9b.
2020-02-13Refactor TestResult object initializationJon Turney
Just add or alter attributes of the TestResult object, rather than many, many invocations of the constructor with very similar arguments. Define helper methods for TestResult to add the results of a step and set the failure reason.
2020-02-11ci: Add Fedora and OpenSUSE imagesDaniel Mensinger
2020-02-07run_project_tests: Fix it to actually workDylan Baker
This move the sorting logic into the TestDef class itself, which simplifies sorting them. Additionally it remove overcomplicated sort logic, because python strings are compared character by character, we don't need to do the split and cast to int, we know that realistically a maximum of 4 characters (the first 4 numbers) are going to be compared in most cases.
2020-02-05tests: Added basic test matrix support for project testsDaniel Mensinger
2020-01-30Merge pull request #6536 from jon-turney/cross-testing-refactorJussi Pakkanen
Refactor CI cross-testing
2020-01-30Bugfix: sanitize_dir: use pathlib to handle case-insensitive filesystems (#6398)Michael Hirsch, Ph.D
2020-01-28Make run_project_tests.py aware of --cross-file optionJon Turney
2020-01-26cmake: Some test improvementsDaniel Mensinger
2020-01-24tests: Fix some test failures on Ubuntun 16.04 CINirbheek Chauhan
* xenial doesn't ship many dependencies, so make them all optional since we don't guarantee that everything will work * cmake/{5,6}: needs stdlib.h for EXIT_SUCCESS on GCC 5 * common/222: needs C++11, and GCC 5 doesn't understand `auto` correctly unless we explicitly enable it. * frameworks/1 boost: xenial doesn't ship boost_python3, so make it properly optional * frameworks/6 gettext: gettext can be installed without xgettext, which doesn't cause the project to fail, but the installed files list is different which causes the test to fail. * frameworks/7 gnome: gobject-introspection can't be enabled because the sanitizer unit test detects leaks in glib and fails
2020-01-08types: import typing as T (fixes #6333)Daniel Mensinger
2019-12-23doc: cuda windows commentMichael Hirsch, Ph.D
2019-12-11Merge pull request #6207 from dcbaker/linker-optionJussi Pakkanen
Add a way to select the dynamic linker meson uses
2019-12-07run_project_tests: argparse(choices) for --only name checkMichael Hirsch, Ph.D
2019-12-07Merge pull request #6228 from scivision/scalapackJussi Pakkanen
deps: add Scalapack
2019-12-05lgtm: fix Unreachable codeDaniel Mensinger
2019-12-03run_project_tests: Fix skip rustDylan Baker
which expects an argument of backend, but isn't getting one.
2019-11-30CI: scalapack not skippable since it's now in Docker imageMichael Hirsch, Ph.D
2019-11-30deps: add scalapackMichael Hirsch, Ph.D
Scalapack uses a library stack that can be challenging to manage. Not least of all since many Scalapacks ship with broken / incomplete pkg-config files and CMake FindScalapack.cmake This resolves those issues for typical Scalapack setups including: * Linux: Intel MKL or OpenMPI + Netlib * MacOS: Intel MKL or OpenMPI + Netlib * Windows: Intel MKL (OpenMPI not available on Windows)