Skip to content

bpo-34279: Issue a warning if no tests have been executed #10150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 29, 2018

Conversation

pablogsal
Copy link
Member

@pablogsal pablogsal commented Oct 27, 2018

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to add a list of tests which ran no tests to the Regrtest class, so it becomes possible to list them in display_result().

It would be better if test_sys is listed in the final report when running: "./python -m test -j0 test_os test_sys -m test_access". Currently, you have to read all logs to notice that test_sys ran no tests.

@bedevere-bot
Copy link

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

And if you don't make the requested changes, you will be poked with soft cushions!

@@ -458,6 +466,9 @@ def get_tests_result(self):
result.append("FAILURE")
elif self.ns.fail_env_changed and self.environment_changed:
result.append("ENV CHANGED")
elif not any((self.good, self.bad, self.skipped, self.interrupted,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of always appending "NO TEST RUN" if run_no_tests is non-empty?

"./python -m test -j0 test_os test_sys -m test_access" would say "Tests result: SUCCESS, NO TEST RUN".

Does it sound weird to you?

My expectation is that "SUCCESS, NO TEST RUN" would be rare. You would only get it if you pass --match. But if you get it, it means that maybe you did a mistake. That's why the exit code is 0 (success) and not 1 (error). It's just a hint, not an hard error.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hummmm..... I think is a good idea. What do you think about somethink on the lines of:

❯ ./python.exe -m test -j0 test_os test_sys -m test_access
Run tests in parallel using 10 child processes
0:00:00 load avg: 1.72 [1/2] test_sys run no tests
0:00:00 load avg: 1.72 [2/2] test_os passed

== Tests result: SUCCESS, (SOME TESTS DID NOT EXECUTE ANY SUBTESTS) ==

1 test OK.

1 test run no tests:
    test_sys

Total duration: 473 ms
Tests result: SUCCESS, (SOME TESTS DID NOT EXECUTE ANY SUBTESTS)

only if the output is SUCCESS? So basically:

if not result:
     result.append("SUCCESS")
     if self.run_no_tests:
         result.append("(SOME TESTS DID NOT EXECUTE ANY SUBTESTS)")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only if the output is SUCCESS

I don't think that it's useful. And I prefer "NO TEST RUN" because it's shorter :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablogsal: what do you think? ping?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late response.

I agree, I think printing "NO TEST RUN" when there is at least one item in run_no_tests is a good idea :)

I implemented it in 8a64c0b

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, I am checking somethig that is not fully correct with that commit.

Copy link
Member Author

@pablogsal pablogsal Nov 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we are setting "SUCCESS" only when there are no other thing in result this means that running ./python -m test -j0 test_os -m test_no_existing will also result in:

Tests result: SUCCESS, NO TEST RUN

as to achieve the result for "./python -m test -j0 test_os test_sys -m test_access" we would need to append after the "SUCCESS" is added:

if not result:
    result.append("SUCCESS")
if self.run_no_tests:
    result.append("NO TEST RUN")

otherwise, you never will get "SUCCESS" when "self.run_no_test" is populated.

I suposse that is not what we want. What do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we are setting "SUCCESS" only when there are no other thing in result this means that running ./python -m test -j0 test_os -m test_no_existing will also result in:

Tests result: SUCCESS, NO TEST RUN

as to achieve the result for "./python -m test -j0 test_os test_sys -m test_access" we would need to append after the "SUCCESS" is added:

if not result:
    result.append("SUCCESS")
if self.run_no_tests:
    result.append("NO TEST RUN")

otherwise, you never will get "SUCCESS" when "self.run_no_test" is populated.

I suposse that is not what we want. What do you think?

Ok, nervermind. I implemented it in 2272ba8 so only Tests result: SUCCESS, NO TEST RUN will happen if there is at least one test that succeeds and some did not run but only NO TEST RUN appears if no test at all have ran. Some examples:

./python -m test -j0 test_os test_sys -m test_access
Tests result: SUCCESS, NO TEST RUN

./python -m test -j0 test_os test_sys -m test_blech
Tests result: NO TEST RUN

./python -m test -j0 test_os -m test_blech
Tests result: NO TEST RUN

./python -m test -j0 test_os -m test_access
Tests result: SUCCESS

@vstinner
Copy link
Member

./python -m test -j0 test_os test_sys -m test_access
Tests result: SUCCESS, NO TEST RUN

This command runs test_os.test_access(), it does not run "no test": it runs one test :-) Maybe we should only display NO TEST RUN is 0 test has been run? So display "SUCCESS" on the first example.

NO TEST RUN would make sense on such example:

./python -m test -j0 test_os test_sys -m test_blech
Tests result: NO TEST RUN

Sorry if I asked something different, I wasn't sure of what I want :-)

"Tests result: NO TEST RUN" means "you did something wrong", whereas "./python -m test -j0 test_os test_sys -m test_access" is not a mistake: it can be deliberate to run 0 test in test_sys. This is especially true when you use "./python -m test.bisect".

@pablogsal
Copy link
Member Author

./python -m test -j0 test_os test_sys -m test_access
Tests result: SUCCESS, NO TEST RUN

This command runs test_os.test_access(), it does not run "no test": it runs one test :-) Maybe we should only display NO TEST RUN is 0 test has been run? So display "SUCCESS" on the first example.

NO TEST RUN would make sense on such example:

./python -m test -j0 test_os test_sys -m test_blech
Tests result: NO TEST RUN

Sorry if I asked something different, I wasn't sure of what I want :-)

"Tests result: NO TEST RUN" means "you did something wrong", whereas "./python -m test -j0 test_os test_sys -m test_access" is not a mistake: it can be deliberate to run 0 test in test_sys. This is especially true when you use "./python -m test.bisect".

That was the previous behaviour :)

Basically, without commit 2272ba8 this is the situation:

❯ ./python -m test -j0 test_os test_sys -m test_access
Tests result: SUCCESS

❯ ./python -m test -j0 test_os test_sys -m test_blech
Tests result: NO TEST RUN

❯ ./python -m test -j0 test_os -m test_blech
Tests result: NO TEST RUN

❯ ./python -m test -j0 test_os -m test_access
Tests result: SUCCESS

I reverted 2272ba8 in 9c7e7b5, so this PR is how it was before it (as in the example in this message).

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but you may move the NEWS entry to the Tests category.

@vstinner vstinner merged commit 9724348 into python:master Nov 29, 2018
vstinner added a commit that referenced this pull request Nov 29, 2018
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099)

Rename also run_tests_slave() to run_tests_worker().

(cherry picked from commit 012f5b9)

* bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150)

(cherry picked from commit 9724348)

* test_regrtest: remove unused threading import
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 29, 2018
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (pythonGH-9099)

Rename also run_tests_slave() to run_tests_worker().

(cherry picked from commit 012f5b9)

* bpo-34279, regrtest: Issue a warning if no tests have been executed (pythonGH-10150)

(cherry picked from commit 9724348)

* test_regrtest: remove unused threading import
(cherry picked from commit 8a73cac)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
miss-islington added a commit that referenced this pull request Nov 29, 2018
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099)

Rename also run_tests_slave() to run_tests_worker().

(cherry picked from commit 012f5b9)

* bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150)

(cherry picked from commit 9724348)

* test_regrtest: remove unused threading import
(cherry picked from commit 8a73cac)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
@pablogsal pablogsal deleted the no_test_run branch May 19, 2021 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants