summaryrefslogtreecommitdiff
path: root/run_project_tests.py
AgeCommit message (Collapse)Author
2023-01-12tests: properly log the reason why a test got unexpectedly skippedEli Schwartz
2022-12-15Enable Java project tests on DarwimTristan Partin
2022-11-23python: Use correct extension filename suffix on Python < 3.8.7Daniele Nicolodi
On Windows, in Python version prior to 3.8.7, the `sysconfig` modules provides an extension filename suffix that disagrees the one returned by `distutils.sysconfig`. Get the more awesome suffix from the latter when building for a Python version known to present this issue. Simplify the extension module filename suffix lookup to use the same method used by `setuptools`. Adjust project tests accordingly. Fixes #10960.
2022-11-22test runner: fix excluding null optionsMarvin Scholz
It was impossible to specify null arguments in the exclude array, while they were passed properly in the code, they were stringified while the entries in the options array were just completely removed. To fix this, do not early stringify the argument name and option together as that would make it impossible to properly match it to None without introducing a new special case for the string "None". So instead they are now kept as tuples and only stringified when its actually needed.
2022-11-17fix deprecated use of meson builddir/ in testsuiteEli Schwartz
2022-10-24basic support for oneapi compilersRobert Cohn
2022-08-17run_project_tests: also change paths for DEPRECATION messagesDylan Baker
Not just WARNING and ERROR
2022-08-16deprecate running "meson builddir" without setup subcommandEli Schwartz
This is ambiguous, if the build directory has the same name as a subcommand then we end up running the subcommand. It also means we have a hard time adding *new* subcommands, because if it is a popular name of a build directory then suddenly scripts that try to set up a build directory end up running a subcommand instead. The fact that we support this at all is a legacy design. Back in the day, the "meson" program was for setting up a build directory and all other tools were their own entry points, e.g. `mesontest` or `mesonconf`. Then in commit fa278f351fe3d6924b4d1961f77b5b4a36e133f8 we migrated to the subcommand mechanism. So, for backwards compatibility, we made those tools print a warning and then invoke `meson <tool>`. We also made the `meson` tool default to setup. However, we only warned for the other tools whose entry points were eventually deleted. We never warned for setup itself, we just continued to silently default to setup if no tool was provided. `meson setup` has worked since 0.42, which is 5 years old this week. It's available essentially everywhere. No one needs to use the old backwards-compatible invocation method, but it continues to drag down our ability to innovate. Let's finally do what we should have done a long time ago, and sunset it.
2022-07-06run_project_tests: Don't try to use tqdm when stdout is not a ttyDylan Baker
Like when piped to something like less, where tqdm just makes a mess of the output instead of providing something helpful.
2022-05-10Require CMake 3.14 for all tests that use the cmake moduleSimon McVittie
Older versions are not supported by the cmake module since 0.62. This avoids having to hard-code the linux-bionic-gcc CI job as being unable to run these tests, which leaves other older environments like Debian 10 still trying to run them (and failing). Signed-off-by: Simon McVittie <smcv@collabora.com>
2022-05-06Limit parallelism to hopefully work on machines with 160 cores.Jussi Pakkanen
2022-05-01another day, another mypy update becomes stricter about typingEli Schwartz
Move GlobalState to a runtime T.NamedTuple, use it for constructing the tuple we are passing around rather than expecting mypy to detect that the one we already have matches.
2022-04-12python module: windows dll name for pypy needs special casingMatti Picus
2022-04-03project tests: log the reason why a test is skippedEli Schwartz
We expose a reason after the string 'MESON_SKIP_TEST', but it is actually ignored when running the test, so it is only useful as documentation and really might as well be a comment. Make it even more useful by actually printing that string after the '[SKIPPED]' message. Also, sometimes a test can be skipped for multiple reasons, and it would be useful to know which one occurred.
2022-04-03project tests: do not use tqdm progress bars for a single testEli Schwartz
This can happen for example with `--only fpga` as that category has exactly one test.
2022-03-29move typing-only definition to TYPE_CHECKINGEli Schwartz
2022-02-27modules/wayland: Add unstable_wayland moduleMark Bolhuis
2022-02-17project tests: fine-tune the check for whether we are in CIEli Schwartz
Follow-up on commit 4274e0f42a9673df981dd7a7a456f4d4fcfe5452. We want to allow tests to be skipped freely in third-party environments, so this should check the jobname, not whether $CI exists. We will anyways raise an error when trying to run the project tests, if $CI is set but no jobname is set.
2022-02-17tests: allow setting MESON_CI_JOBNAME=thirdpartyEli Schwartz
This is treated by the test harness as though unset, i.e. we do normal skipping and don't assume we are running in Meson's own project CI. However, it has one distinction which is that it isn't an error to set $CI without setting $MESON_CI_JOBNAME, if it is in fact set but to the ignored value. This lets automated workflows such as Linux distro testing, particularly alpine linux, set $CI or have it set for them by default, without messing things up. Also it has the advantage of $CI actually enabling useful benefits! We will still assume that this thirdparty environment wants to force verbose logging (printing testlogs, running ninja/samu with -v) and colorize the console.
2022-02-03cmake: ci: Skip tests on Ubuntu Bionic where CMake is stuck on 3.10Daniel Mensinger
2022-01-10pyupgrade --py37-plusEli Schwartz
2021-12-17Fix mypy errorsDaniel Mensinger
2021-10-10Add --vsenv command line option and active VS only when neededXavier Claessens
2021-10-04various python neatness cleanupsEli Schwartz
All changes were created by running "pyupgrade --py3-only" and committing the results. Although this has been performed in the past, newer versions of pyupgrade can automatically catch more opportunities, notably list comprehensions can use generators instead, in the following cases: - unpacking into function arguments as function(*generator) - unpacking into assignments of the form x, y = generator - as the argument to some builtin functions such as min/max/sorted Also catch a few creeping cases of new code added using older styles.
2021-09-30Fix typosChristian Clauss
2021-09-10Presence of nagfor signifies that Fortran tests can be run.Mat Cross
2021-08-22python: Workaround broken install pathXavier Claessens
2021-08-18test runner: add the ability to configure test.json for python pathsEli Schwartz
Adds new installed file types with @VAR@ substitution.
2021-08-17Revert "Disable Rust + clangcl."Nirbheek Chauhan
This reverts commit 6c55fc5cb0f6f363d5221c9bd0c1fd1fc6f90f4a
2021-08-18Revert "Disable vs2017 + 32bit Rust combo as it is broken."Nirbheek Chauhan
This reverts commit 75688240cfca7eed08c2754daa784c9bd1a70a73 Should've been fixed by c95bffb295dd8ae12dd37c9a6c33372a20cf9a68
2021-08-15tests: make detecting broken compilers always run without tracebacksEli Schwartz
Fixes regression in commit 75688240cfca7eed08c2754daa784c9bd1a70a73. Even though this function is *currently* only invoked on Windows, these environment variables may not actually exist -- and apparently don't in at least the "UnusedMissingReturn / windows" test run, which... did not get triggered by that commit, since it only edited the testsuite runner, not any test cases. \o/
2021-08-15Disable Rust + clangcl.Jussi Pakkanen
2021-08-15Disable vs2017 + 32bit Rust combo as it is broken.Jussi Pakkanen
2021-08-12Allow gettext test to skip on macOS.Jussi Pakkanen
2021-07-25Remove workaround now that we have a new Ninja.Jussi Pakkanen
2021-07-13Make it an error if MESON_CI_JOBNAME isn't set under CIJon Turney
Also check skip is as expected for CI all runs, not just those that set MESON_CI_JOBNAME.
2021-07-07Use None as Environment object build_dir in detect_system_compiler()Jon Turney
The Environment object constructor accepts None as build_dir (for quite a while now), so don't bother with creating a temporary directory for use as the build_dir, if we're not going to need it. Future work: Environment.__init__() sets scratch_dir to '' if build_dir is None, which seems a little wonky, as it isn't a path.
2021-07-07Annotate framework tests with where they are expected to skipJon Turney
Remove hard-coded framework test skip logic in skippable(), instead annotate test.json with environments in which skip is expected. (Mainly this is done with by testing the value of MESON_CI_JOBNAME now set for linux jobs)
2021-07-07Allow `skip_*` test.json keys at top-level or in `matrix:`Jon Turney
2021-07-07Add `skip_on_os` to test.jsonJon Turney
2021-07-07Add `skip_on_jobname` to test.jsonJon Turney
Plan to replace the hard-coded list of 'may be skipped' framework tests in skippable() with annotations in test.json which record 'will be skipped in these specific CI jobs'. If the value of the MESON_CI_JOBNAME env var (an arbitrary string expected to be unique for each CI configuration) contains any of the strings in the `skip_on_jobname` key in test.json, the test is expected to output MESON_SKIP_TEST. Unexpected skips or runs are treated as an error. Future work: Maybe we should add additional count categories 'unexpected skip' and 'unexpected not skipped', rather than counting those as 'skipped' and 'failed', respectively.
2021-07-07Drop non-framework test names from skippable()Jon Turney
In all these cases, the test has already been determined as skippable because it's not part of the 'frameworks' suite.
2021-07-05pyupgradeEli Schwartz
2021-06-29fix: Always explicitly set encoding for text files (fixes #8263)Daniel Mensinger
2021-06-29pathlib: Patch pathlib to work around some bugs (fixes #7295)Daniel Mensinger
2021-06-25Split compiler detection from EnvironmentDaniel Mensinger
This moves all the compiler detection logic into the new compilers.detect module. This dramatically reduces the size and complexity of Environment.
2021-06-22Fix project tests category name used in skippable() and XML outputJon Turney
Since 25df6e7d split the iteration over tests to start them from the iteration to collect their results, the variable 'name' is only being set in the first iteratiorn, so all tests are treated as being in the last test category read (probably 'wasm') for skipppable() and in the XML output. Store the category name in the TestDef object Use it in skippable() Use it in classname attribute of XML test results
2021-06-18tests: Always enable the traceback in run_project_tests.pyDaniel Mensinger
2021-06-18tests: Add visual seperator to the test logs in run_project_tests.pyDaniel Mensinger
2021-06-11CI: Split out file format test as a separate GitHub workflowJon Turney
Split out checking of file format as a separate GitHub workflow, rather than running it as part of the project tests for every platform and toolchain combination in CI, so that this test is not effected by the changed paths constraints which are applied to the project tests.