-
-
Notifications
You must be signed in to change notification settings - Fork 32.2k
bpo-34279: Issue a warning if no tests have been executed #10150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
dc0da37
to
c684dd4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest to add a list of tests which ran no tests to the Regrtest class, so it becomes possible to list them in display_result().
It would be better if test_sys is listed in the final report when running: "./python -m test -j0 test_os test_sys -m test_access". Currently, you have to read all logs to notice that test_sys ran no tests.
When you're done making the requested changes, leave the comment: And if you don't make the requested changes, you will be poked with soft cushions! |
@@ -458,6 +466,9 @@ def get_tests_result(self): | |||
result.append("FAILURE") | |||
elif self.ns.fail_env_changed and self.environment_changed: | |||
result.append("ENV CHANGED") | |||
elif not any((self.good, self.bad, self.skipped, self.interrupted, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think of always appending "NO TEST RUN" if run_no_tests is non-empty?
"./python -m test -j0 test_os test_sys -m test_access" would say "Tests result: SUCCESS, NO TEST RUN".
Does it sound weird to you?
My expectation is that "SUCCESS, NO TEST RUN" would be rare. You would only get it if you pass --match. But if you get it, it means that maybe you did a mistake. That's why the exit code is 0 (success) and not 1 (error). It's just a hint, not an hard error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hummmm..... I think is a good idea. What do you think about somethink on the lines of:
❯ ./python.exe -m test -j0 test_os test_sys -m test_access
Run tests in parallel using 10 child processes
0:00:00 load avg: 1.72 [1/2] test_sys run no tests
0:00:00 load avg: 1.72 [2/2] test_os passed
== Tests result: SUCCESS, (SOME TESTS DID NOT EXECUTE ANY SUBTESTS) ==
1 test OK.
1 test run no tests:
test_sys
Total duration: 473 ms
Tests result: SUCCESS, (SOME TESTS DID NOT EXECUTE ANY SUBTESTS)
only if the output is SUCCESS? So basically:
if not result:
result.append("SUCCESS")
if self.run_no_tests:
result.append("(SOME TESTS DID NOT EXECUTE ANY SUBTESTS)")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only if the output is SUCCESS
I don't think that it's useful. And I prefer "NO TEST RUN" because it's shorter :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pablogsal: what do you think? ping?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the late response.
I agree, I think printing "NO TEST RUN" when there is at least one item in run_no_tests
is a good idea :)
I implemented it in 8a64c0b
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait, I am checking somethig that is not fully correct with that commit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we are setting "SUCCESS" only when there are no other thing in result
this means that running ./python -m test -j0 test_os -m test_no_existing
will also result in:
Tests result: SUCCESS, NO TEST RUN
as to achieve the result for "./python -m test -j0 test_os test_sys -m test_access"
we would need to append after the "SUCCESS" is added:
if not result:
result.append("SUCCESS")
if self.run_no_tests:
result.append("NO TEST RUN")
otherwise, you never will get "SUCCESS" when "self.run_no_test" is populated.
I suposse that is not what we want. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we are setting "SUCCESS" only when there are no other thing in
result
this means that running./python -m test -j0 test_os -m test_no_existing
will also result in:Tests result: SUCCESS, NO TEST RUN
as to achieve the result for
"./python -m test -j0 test_os test_sys -m test_access"
we would need to append after the "SUCCESS" is added:if not result: result.append("SUCCESS") if self.run_no_tests: result.append("NO TEST RUN")
otherwise, you never will get "SUCCESS" when "self.run_no_test" is populated.
I suposse that is not what we want. What do you think?
Ok, nervermind. I implemented it in 2272ba8 so only Tests result: SUCCESS, NO TEST RUN
will happen if there is at least one test that succeeds and some did not run but only NO TEST RUN
appears if no test at all have ran. Some examples:
./python -m test -j0 test_os test_sys -m test_access
Tests result: SUCCESS, NO TEST RUN
./python -m test -j0 test_os test_sys -m test_blech
Tests result: NO TEST RUN
./python -m test -j0 test_os -m test_blech
Tests result: NO TEST RUN
./python -m test -j0 test_os -m test_access
Tests result: SUCCESS
8a64c0b
to
367cfd1
Compare
367cfd1
to
2272ba8
Compare
This command runs test_os.test_access(), it does not run "no test": it runs one test :-) Maybe we should only display NO TEST RUN is 0 test has been run? So display "SUCCESS" on the first example. NO TEST RUN would make sense on such example:
Sorry if I asked something different, I wasn't sure of what I want :-) "Tests result: NO TEST RUN" means "you did something wrong", whereas "./python -m test -j0 test_os test_sys -m test_access" is not a mistake: it can be deliberate to run 0 test in test_sys. This is especially true when you use "./python -m test.bisect". |
This reverts commit 2272ba8.
That was the previous behaviour :) Basically, without commit 2272ba8 this is the situation:
I reverted 2272ba8 in 9c7e7b5, so this PR is how it was before it (as in the example in this message). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, but you may move the NEWS entry to the Tests category.
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099) Rename also run_tests_slave() to run_tests_worker(). (cherry picked from commit 012f5b9) * bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150) (cherry picked from commit 9724348) * test_regrtest: remove unused threading import
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (pythonGH-9099) Rename also run_tests_slave() to run_tests_worker(). (cherry picked from commit 012f5b9) * bpo-34279, regrtest: Issue a warning if no tests have been executed (pythonGH-10150) (cherry picked from commit 9724348) * test_regrtest: remove unused threading import (cherry picked from commit 8a73cac) Co-authored-by: Victor Stinner <vstinner@redhat.com>
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099) Rename also run_tests_slave() to run_tests_worker(). (cherry picked from commit 012f5b9) * bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150) (cherry picked from commit 9724348) * test_regrtest: remove unused threading import (cherry picked from commit 8a73cac) Co-authored-by: Victor Stinner <vstinner@redhat.com>
https://bugs.python.org/issue34279