| Age | Commit message (Collapse) | Author |
|---|
|
The previous commit eb1e52afa142fc0f38260a9cb3413f2bd63b1675 introduced
a variable `rebuild_only_tests` to fix various edge cases in
rebuild_deps.
In particular, if all tests are selected anyway, rebuild_deps prior
to the introduction of `rebuild_only_tests` would potentially
create a huge list of targets, which may overflow the
ARG_MAX limit.
For that case `rebuild_only_tests` was introduced and is set to an empty list, which means that rebuild_deps falls back to the `meson-test-prereq` ninja target, that already contains the necessary targets.
This is wrong, though, when `meson test` is executed with the "--suite"
option. In that case, the user requested a particular subset of the
tests, but `rebuild_only_tests` will be set to an empty list anyway,
which means the `meson-test-prereq` target is executed, which contains
the targets for all tests.
Instead, `rebuild_only_tests` should only be set to an empty list,
when the selected tests are identical to the complete list of available
tests: `tests == self.tests`
Since after this commit we compare directly to the result of `self.get_tests()`, this
will do the correct thing for all other options, which change or filter
the list of selected tests (e.g. `self.options.args`, `self.options.slice`).
|
|
Previously, we exited as "number of test failures", as a side effect of
a refactor. Regressed in commit 297749581dd4dcb4e68da4b1d0622d2916fb8db3.
This was a bad sentinel for "track if tests have failed", and formerly
we did indeed set "returncode = 1" upon encountering a test fail! The
consequence of this is that we break:
- returning failure at all, if test fails are a multiple of 256
- returning 125 for "git bisect run shall skip this commit", as a
documented guarantee of meson
- returning > 127 shall abort bisection entirely
The behavior is nonsensical and accidental, and nobody should be using
this accident to count the number of failed tests (as it won't even work).
Fixes: https://github.com/mesonbuild/meson/issues/15251
|
|
This info is also available from the "result" entry, but requires
mapping result types to "would cause a failure, but isn't literally
FAIL". It comes in handy for the meson testsuite, too.
|
|
The entire policy system has been deprecated in Python 3.14, and is
slated for removal in 3.16. This has been caught by pylint, and is
causing CI to fail.
For our use case this is only relevant on Python 3.7 on Windows, as the
default policy on 3.8 is set to the Proactor anyway.
|
|
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
This makes easier to run a single test after running them all. For
example the test "test_accsadubl" in the suite "default" of project
"orc" used to be displayed "orc:default / test_accsadubl". To run that
test specifically the command line is `meson test -C builddir
orc:test_accsadubl --suite default`. To make this more consistent
the test is now displayed as "default - orc:test_accsadubl".
|
|
Omitting test()'s `suite` kwarg used to default to `test.suite==['']`
because of how stringlistify() works. That had inconsistent behavior
with `suite: []` which makes `test.suite==[]`. That changed when porting
to typed_kwarg (#8855), which made the default `test.suite==[]`.
Change `suite: []`, and missing kwarg, back to `test.suite==['']` which
was the original intention. It has impact on `meson test --suite`
selection, an empty `test.suite` list means that test is never going to
be selected when `--suite` is used, even when we want all tests from a
given subproject.
It was also unclear what `--suite foo` means. Is `foo` a suite or a
project name? It can now be either, it selects all tests from project
`foo` as well as all tests from any project in suite `foo`.
|
|
|
|
--wrapper and --gdb are mutually exclusive per run(), so make get_wrapper
reflect this (which would've saved me some time earlier with trying to
make the two work together for something else, when it turns out that's
impossible).
As suggested by Eli.
|
|
We currently suggest that users run `meson test --wrapper valgrind`, but
this doesn't do what one might expect: Valgrind doesn't error out on
violations/issues it detects.
In the past, we had special handling for Valgrind in tests, see
1f76b76a84cb635f764ecbd2b77aaba1d375d72b but it was later dropped in
951262d7590343ffa9730666c427ad9d708a9fb6.
This is similar to what we do for {A,UB,M}SAN_OPTIONS to give sensible
behaviour that users expect out-of-the-box.
Only do this if we see 'valgrind' in the wrapper command to avoid
polluting logs. We may want to do that for the sanitizers variables
in future too.
Note that we're not adding --exit-on-first-error=yes here, as there
may be several issues in an application, or a test may be rather slow,
and so on. But --error-exitcode=1 to simply make Valgrind's exit status
reflect whether an error was found is uncontroversial.
Bug: https://github.com/mesonbuild/meson/issues/4727
Bug: https://github.com/mesonbuild/meson/issues/1105
Bug: https://github.com/mesonbuild/meson/issues/1175
Bug: https://github.com/mesonbuild/meson/issues/13745
|
|
This is an old method, that is now just a wrapper around the OptionStore
method, that doesn't add any value. It's also an option related method
attached to the CoreData instead of the OptionStore, so useless and a
layering violation.
|
|
When running tests with `--interactive` we don't redirect stdin, stdout
or stderr and instead pass them on to the user's console. This redirect
causes us to hang in case the test in question needs parsing, like it is
the case for TAP output, because we cannot read the process's stdout.
Fix this hang by not parsing output when running in interactive mode.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
|
|
When running tests in interactive mode then the standard file streams
will remain connected to the executing terminal so that the user can
interact with the tests. This has the consequence that Meson itself does
not have access to those streams anymore, which is problematic for any
of the test types that require parsing, like for example with the TAP
protocol. This means that Meson is essentially flying blind in those
cases because the test result cannot be determined by parsing the exit
code of the test, but can only reliably be derived from the parsed
output.
One obvious solution to this problem would be to splice the test output
so that both Meson and the user's terminal have access to it. But when
running in interactive mode it is quite likely that the test itself will
actually be driven from the command line, and the chance is high that
the resulting data on stdout cannot be parsed as properly anymore. This
is for example the case in the Git project, where interactive mode is
typically used to drop the user into a shell or invoke a debugger.
So continuing to treat the output as properly formatted output that can
be parsed is likely a dead end in many use cases. Instead, we introduce
a new "IGNORED" test result: when executing tests in interactive mode,
and when the test type indicates that it requires parsing, we will not
try to parse the test at all but mark the test result as ignored
instead.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
|
|
Move the `console_mode` property into the TestRun Class. This will be
required by a subsequent commit where we start to ignore test results
for parsed interactive tests.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
|
|
After a test run finishes we print a summary that sums up test counts by
type, e.g. failed or skipped tests. In many cases though it is expected
that most of the counts will be zero, and thus the summary is needlessly
cluttered with irrelevant lines. This list of mostly-irrelevant results
will grow in a subsequent commit where we introduce "Ignored" test
results.
Prepare for this by filtering results. Instead of unconditionally
printing every result, we will now only print those results where we
have at least one counted test. The exception is "Ok:" and "Fail:",
which will always be printed.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
|
|
Executing tests can take a very long time. As an example, the Git test
suite on Windows takes around 4 hours to execute. The Git project has
been working around the issue by splitting up CI jobs into multiple
slices: one job creates the build artifacts, and then we spawn N test
jobs with those artifacts, where each test job executes 1/Nth of the
tests.
This can be scripted rather easily by using `meson test --list`,
selecting every Nth line, but there may be other projects that have a
similar need. Wire up a new option "--slice i/n" to `meson test` that
does implements this logic.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
|
|
|
|
Inconsistency in the original implementation of commit
79e2c52a15e896e46ff3cfa3ec16fbf3f132ee01.
If an explicit list of targets is passed on the CLI, then that is passed
to rebuild_deps. If not, we pass every loaded test to rebuild_deps
instead. This means we cannot distinguish between "trying to run all
tests" and "trying to run specific tests". We then load all the deps for
all tests, and try to build them all as explicit arguments to the
underlying ninja.
There are two situations where this falls flat:
- given underspecified deps
- given all (selected?) tests legitimately happen to have no
dependencies
In both cases, we calculate that there are no deps to rebuild, we run
ninja without any targets, and this invokes the default "all" rule and
maybe builds a few thousand targets that this specific test run does not
need.
Additionally, in large projects which define many tests with many
dependencies, we could end up overflowing ARG_MAX when processing *all*
tests.
Instead, pass no tests to rebuild_deps. We then specially handle this by
directly running the relevant ninja target for "all test deps", which is
overall more elegant than specifying many many dependencies by name.
Given a subset of tests to guarantee the freshness of, we instead skip
running ninja at all if there are indeed no test dependencies.
|
|
Resolves: #13802
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
It is useful to apply a limit to the number of processes even outside "meson test",
and specifically for clang tools. In preparation for this, generalize
determine_worker_count() to accept a variable MESON_NUM_PROCESSES instead of
MESON_TESTTHREADS, and use it throughout instead of multiprocessing.cpu_count().
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
import mintro and its attendant module dependency tree just
so we can programmatically get filenames which are documented
as a stable API in https://mesonbuild.com/IDE-integration.html.
Suggested-by: Eli Schwartz <eschwartz93@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Doctests have a slightly different output compared to what "protocol: rust"
supports:
running 2 tests
test ../doctest1.rs - my_func (line 7) ... ignored
test ../doctest1.rs - (line 3) ... ok
test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.12s
Add a little more parsing in order to accept this; a simple minded split()
fails to unpack the tuple. I plan to contribute an extension of the rust
module to invoke doctests, for now this allows running rustdoc --test with
"protocol: 'rust'" and get information about the subtests:
▶ 4/8 ../doctest1.rs:my_func:7 SKIP
▶ 4/8 ../doctest1.rs:3 OK
4/8 rust_unit_tests:doctests / rust doctest OK 0.28s 1 subtests passed
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
|
|
Previously, setting `MESON_TESTTHREADS` to a number lower than 1
resulted in unexpected behavior. This commit introduces test for
negative value (with fallback to 1), and fallback to core count in case
it is set to 0.
It improves experience in job-matrix type of CI workflows, where some
jobs within the matrix require single job execution, whereas others can
default to taking core count as the job count.
Signed-off-by: Marek Pikuła <m.pikula@partner.samsung.com>
|
|
On Windows, the output read from the stream has '\r\n', which in .txt,
.json and console logger (when captured to a file) translates to '\n\n'.
This results in every log line being separated by an empty line.
|
|
|
|
Let's allow users to configure how many lines are shown at most when
a test fails.
|
|
When running our integration tests in systemd we depend on each test
having a unique name. This is always the case unless --repeat is used,
in which case multiple tests with the same name run concurrently which
causes issues when allocating resources that use the test name as the
identifier.
Let's set MESON_TEST_ITERATION to the current iteration of the test so
we can use $TEST_NAME-$TEST_ITERATION as our test identifiers which will
avoid these issues.
|
|
This allows tests to check whether stdin is a tty to figure out if they're
running in interactive mode or not.
It also makes sure that tests that are not running in interactive mode
don't inadvertendly try to read from stdin.
|
|
This is very similar to --gdb, except it doesn't spawn GDB, but
connects stdin/stdout/stderr directly to the test itself. This allows
interacting with integration tests that spawn a shell in a container
or virtual machine when the test fails.
In systemd we're migrating our integration tests to run using the
meson test runner. We want to allow interactive debugging of failed
tests directly in the virtual machine or container that is spawned
to run the test. To make this possible, we need meson test to connect
stdin/stdout/stderr of the test directly to the user's terminal, just
like is done with the --gdb option.
|
|
|
|
Fixes #6567
|
|
Followup to 7b7d2e060b447de9c2642848847370a58711ac1c which handles ASAN and UBSAN.
It turns out that MSAN needs the same treatment. I've checked other sanitizers
like HWASAN and TSAN - it looks like they may both need it too, but Meson doesn't
currently suppose those anyway (see https://github.com/mesonbuild/meson/pull/12648).
Signed-off-by: Sam James <sam@gentoo.org>
Signed-off-by: Eli Schwartz <eschwartz93@gmail.com>
|
|
Oops.
Fixes: 7b7d2e060b447de9c2642848847370a58711ac1c
Signed-off-by: Sam James <sam@gentoo.org>
|
|
This replaces all of the Apache blurbs at the start of each file with an
`# SPDX-License-Identifier: Apache-2.0` string. It also fixes existing
uses to be consistent in capitalization, and to be placed above any
copyright notices.
This removes nearly 3000 lines of boilerplate from the project (only
python files), which no developer cares to look at.
SPDX is in common use, particularly in the Linux kernel, and is the
recommended format for Meson's own `project(license: )` field
|
|
It is already done by RealPathAction and mypy started complaining about
it.
|
|
line argument code
|
|
We already use -j to support parallelism in meson compile. So let's add
the same for meson test and meson subprojects.
|
|
Do as we do for MALLOC_PERTURB and set a sensible value for both ASAN_OPTIONS
and UBSAN_OPTIONS to abort on failure and give more helpful output at the
same time. We do not set these options if the user has exported a value
themselves to allow override.
In the last week alone, I've observed two cases where people were expecting
sanitizers to abort on failure and were surprised when it didn't:
1) https://github.com/git/git/commit/252d693797912ddb2684733160170f0408b73274
2) https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/c47df433f7bc9936fc59b323ca5e53ea4a04f40e
Correct this - which is in-line with meson's DWIM/DTRT philosophy.
Signed-off-by: Sam James <sam@gentoo.org>
|
|
This was an unintended consequence of the original patch in #11977.
Co-authored-by: Benoit Pierre <benoit.pierre@gmail.com>
|
|
Performed using https://github.com/ilevkivskyi/com2ann
This has no actual effect on the codebase as type checkers (still)
support both and negligible effect on runtime performance since
__future__ annotations ameliorates that. Technically, the bytecode would
be bigger for non function-local annotations, of which we have many
either way.
So if it doesn't really matter, why do a large-scale refactor? Simple:
because people keep wanting to, but it's getting nickle-and-dimed. If
we're going to do this we might as well do it consistently in one shot,
using tooling that guarantees repeatability and correctness.
Repeat with:
```
com2ann mesonbuild/
```
|
|
Make them into real type annotations. These are the only ones that if
automatically rewritten, would cause flake8 to error out with the
message: "E128 continuation line under-indented for visual indent".
|
|
These annotations all had a default initializer of the correct type, or
a parent class annotation.
|
|
This detects cases where module A imports a function from B, and C
imports that same function from A instead of B. It's not part of the API
contract of A, and causes innocent refactoring to break things.
|
|
The original point of specifying Optional was to default to None...
oops. The practical effect of this change is that the testsuite no
longer repeatedly logs "No tests defined." in between more meaningful
output.
|
|
Replace unencodable XML chars with their printable representation, so
that, xmllint can parse test outputs without error.
Closes #9894
Co-authored-by: Tristan Partin <tristan@partin.io>
|
|
Since people may parse the output of `--list` as a list of tests,
putting logging info in stderr is nicer.
|
|
It is not the primary purpose of mtest, and it ends up mingled with a
list of actual tests, which isn't nice if you're trying to capture and
parse this.
|
|
In commit 628effb3698e85e403351d2705d576cf4ee8c50c we started verifying
the build.ninja file was up to date before even `--list`ing tests,
because we could end up with incorrect information. This meant that
ninja always runs at startup, and typically returns "no work to do",
which ended up listed as "one of" the tests.
Instead of unconditionally running ninja attached to the console, first
check it in dry-run mode with stdout intercepted, to see if ninja
considers itself up to date. If it is, continue. Only if an actual
refresh is needed, do we run it while attached to the console.
In the garden path, this avoids useless information. In cases where we'd
already print a full meson reconfigure log, we continue to do so.
|
|
Allow the use of wildcards (e.g. *) to match test names in `meson test`.
Raise an error is given test name does not match any test.
Optimize the search by looping through the list of tests only once.
|
