gh-75117: IDLE - add script for running coverage on tests #22694

taleinat · 2020-10-14T14:17:11Z

This adds a cross-platform script for generating coverage info for any single IDLE module or for all of IDLE.

The script uses a dedicated venv in Lib/idlelib/test_idle/coverage_venv/, creating it if not found. The coverage package is installed into this venv.

The script also writes a temporary .coveragerc file to ignore irrelevant code. If an existing .coveragerc exists, it is temporarily overwritten.

This is heavily based on @terryjreedy's instructions in Lib/idlelib/test_idle/README, adapted to be cross-platform and hopefully robust in terms of where it is run from and where it leaves files lying around.

This has currently only been tested on Ubuntu 20.04, and should be tested on macOS and Windows.

Issue: Document how to run coverage for repository idlelib files. #75117

E-Paine · 2020-10-14T15:03:07Z

I haven't looked properly at it yet, but:

You have trailing whitespace in the run_coverage file
Is it worth modifying test_idle/README.txt for the new script?

taleinat · 2020-10-14T17:32:10Z

You have whitespace in the run_coverage file

Where? Could you leave a specific comment on the relevant lines?

Is it worth modifying test_idle/README.txt for the new script?

Yes, if we agree that this is the way to go, and on some details such as the file name and location.

Lib/idlelib/idle_test/run_coverage.py

taleinat · 2020-10-14T21:29:08Z

@E-Paine, many thanks for the review and great comments! 🙏

taleinat · 2020-10-14T22:16:09Z

@terryjreedy, this is now definitely ready for your review :)

terryjreedy · 2020-10-15T06:44:24Z

I was looking for something to add to, not replace, a revised version of what I already wrote. So I probably will want this patch to leave README alone so I can deal with it in a followup PR. But I will defer that until we finish the sidebar patch.

github-actions · 2020-12-17T00:29:39Z

This PR is stale because it has been open for 30 days with no activity.

erlend-aasland · 2022-07-28T19:34:38Z

Looks good after first read-through. Should mention that this only works with in-tree builds.

Lib/idlelib/idle_test/run_coverage.py

erlend-aasland · 2022-08-03T09:59:10Z

This has currently only been tested on Ubuntu 20.04, and should be tested on macOS and Windows.

Tested on¹:

macOS 12.2.1²: ✔️
Windows 10³: ✔️

Rebased onto main ↩
All but the implicit open-in-browser work swell ↩
open-in-browser needs absolute path; else ok ↩

erlend-aasland · 2022-08-03T10:01:32Z

(Updating branch to trigger the new CLA check)

taleinat · 2022-08-03T12:45:09Z

Thanks for the review @erlend-aasland!

Feel free to remove the parts that open the report in a browser, and instead just print the path to the HTML file.

We should likely get @terryjreedy's blessing before merging this, though.

(Something is very wrong with the CLA bot if it doesn't know that I've signed the CLA...)

erlend-aasland · 2022-08-03T12:47:39Z

Feel free to remove the parts that open the report in a browser, and instead just print the path to the HTML file.

Since webbrowser seems to be broken, or at least problematic, on macOS, the most consistent thing to do is just to remove that functionality. I can update the PR.

(Something is very wrong with the CLA bot if it doesn't know that I've signed the CLA...)

@ambv ☝🏻

ambv · 2022-08-03T14:09:10Z

@taleinat sorry for the mishap. You used to use taleinat+python@gmail.com, now you started using simply taleinat@gmail.com. The CLA bot is dumb at the moment about + and .. There's an open issue about it, I'll address this today and use your case as a test.

erlend-aasland · 2022-08-03T16:22:42Z

Thanks, Łukasz!

terryjreedy

I think coveragerc should be extracted to a file first. I can do it easier since I can just copy my current file.

terryjreedy · 2022-08-03T19:31:25Z

Lib/idlelib/idle_test/README.txt

-(There were several before modules were renamed, now only one is left.)
+To get a coverage report for a specific module's tests, run:
+
+python Lib/idlelib/idle_test/run_coverage.py <module_name>


cd .. cover tooltip

shows me the htmlcov/index.html in about a second, faster than I can type the above. How long does the above run with tooltip?

terryjreedy · 2022-08-03T19:43:16Z

Lib/idlelib/idle_test/README.txt

+To get a coverage report for IDLE's entire test suite, run the
+above command with "all" instead of a module name.


Which of two possible meanings of 'all' is used?

Run each test_x separately and report the coverage for each module the same as if only that test file were run.

Run all tests together and report the coverage for each module that result from running all tests (as well as the collective coverage).
I believe using coverage option -L gives me the second.

terryjreedy · 2022-08-03T19:44:57Z

Lib/idlelib/idle_test/README.txt

+To get a coverage report for IDLE's entire test suite, run the
+above command with "all" instead of a module name.
+
+Note: this does not work with out-of-tree builds yet.


Do you mean installed python or something else?

This PR adds testing for the IDLE test suite in the source directory. I'm one of those folks who like to keep my git clone clean, so I use a build directory outside of the repo directory (AKA an out-of-tree build).

$ cd cpython.git $ git clean -fdx $ mkdir ../build $ cd ../build $ ../cpython.git/configure && make $ ./python.exe ../cpython.git/Lib/idlelib/idle_test/run_coverage.py # <= will not work

$ cd cpython.git $ git clean -fdx $ ./configure && make $ ./python.exe Lib/idlelib/idle_test/run_coverage.py # <= ok

I guess you could use an installed Python to check the coverage of the IDLE test suite in your development environment. I'm not sure why you would want to do so, though.

terryjreedy · 2022-08-03T20:06:06Z

Lib/idlelib/idle_test/run_coverage.py

+
+                    .*# htest #


The readme is behind the file I actually use.

Suggested change

.*# htest #

.*# pragma: no cover

.*# pragma: no branch

.*# htest #

if not (_htest or _utest):

The pragmas are generic to coverage; 'no cover' is currently used in idlelib.

The suggested coveragerc should be a separate file in idle_test/. If I knew how to put a symlink in /dev, where I use it, I would to avoid the 2 copies issues.

terryjreedy · 2022-08-03T20:21:38Z

Lib/idlelib/idle_test/README.txt

---
-The second parameter was added for tests of module x not named test_x.
-(There were several before modules were renamed, now only one is left.)
+To get a coverage report for a specific module's tests, run:


To clarify what I meant 2 years ago, DO NOT REMOVE the instructions for what I have done since 3.6 and plan to continue to use. What I deferred then was revising my text. I am working on it now.

Suggested change

To get a coverage report for a specific module's tests, run:

To get a coverage report for a specific module's tests, do one of the following:

A. run

I can then edit my part to starts B. ....

bedevere-bot · 2022-08-03T20:23:22Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

ambv · 2022-08-04T15:41:34Z

@taleinat, all's good now.

bpo-30934: add script for running coverage on IDLE modules' tests

14ef1b1

taleinat added tests Tests in the Lib/test dir skip news labels Oct 14, 2020

taleinat requested a review from terryjreedy October 14, 2020 14:17

the-knights-who-say-ni added the CLA signed label Oct 14, 2020

bedevere-bot added the awaiting core review label Oct 14, 2020

E-Paine reviewed Oct 14, 2020

View reviewed changes

Lib/idlelib/idle_test/run_coverage.py Outdated Show resolved Hide resolved

Lib/idlelib/idle_test/run_coverage.py Outdated Show resolved Hide resolved

taleinat added 2 commits October 14, 2020 23:18

update README and add support for running all tests

50c9515

refactoring

7d75fb9

E-Paine reviewed Oct 14, 2020

View reviewed changes

taleinat added 6 commits October 15, 2020 00:11

avoid using sys.executable

9d6f38a

call ensure_venv only after parsing cmdline args

f3c644b

add option to avoid generating and opening an HTML report

b137f35

use webbrowser.open() to open HTML reports for viewing

1bd4a55

add a shebang line and make the script executable

fbb51ae

add check for Python version >=3

8110f99

minor tweaks to .gitignore and README

b70b932

github-actions bot added the stale Stale PR or inactive for long period of time. label Dec 17, 2020

ezio-melotti removed the CLA signed label Jul 13, 2022

github-actions bot removed the stale Stale PR or inactive for long period of time. label Jul 30, 2022

erlend-aasland mentioned this pull request Aug 3, 2022

Document how to run coverage for repository idlelib files. #75117

Open

erlend-aasland reviewed Aug 3, 2022

View reviewed changes

Lib/idlelib/idle_test/run_coverage.py Outdated Show resolved Hide resolved

Merge branch 'main' into bpo-30934/idle-test-coverage-script

7fe2ce8

erlend-aasland added 3 commits August 3, 2022 14:50

Mention that out-of-tree builds are not supported

2ecafbe

Remove auto-opening of coverage report

41e9751

Inline it

ce748cc

terryjreedy requested changes Aug 3, 2022

View reviewed changes

bedevere-bot removed the awaiting core review label Aug 3, 2022

bedevere-bot added the awaiting changes label Aug 3, 2022

terryjreedy changed the title ~~bpo-30934: add script for running coverage on IDLE modules' tests~~ gh-75117: add script for running coverage on IDLE modules' tests Aug 3, 2022

ambv closed this Aug 4, 2022

ambv reopened this Aug 4, 2022

ambv mentioned this pull request Aug 4, 2022

Modify email normalization to strip + and . geldata/cla-bot#53

Merged

terryjreedy changed the title ~~gh-75117: add script for running coverage on IDLE modules' tests~~ gh-75117: IDLE - add script for running coverage on tests Nov 2, 2022

python deleted a comment Apr 7, 2025

		To get a coverage report for IDLE's entire test suite, run the
		above command with "all" instead of a module name.

-                    .*# htest #
+                    .*# pragma: no cover
+                    .*# pragma: no branch
+                    .*# htest #
+                    if not (_htest or _utest):

Uh oh!

gh-75117: IDLE - add script for running coverage on tests #22694

Are you sure you want to change the base?

gh-75117: IDLE - add script for running coverage on tests #22694

Conversation

taleinat commented Oct 14, 2020 • edited by terryjreedy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

E-Paine commented Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taleinat commented Oct 14, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taleinat commented Oct 14, 2020

Uh oh!

taleinat commented Oct 14, 2020

Uh oh!

terryjreedy commented Oct 15, 2020

Uh oh!

github-actions bot commented Dec 17, 2020

Uh oh!

erlend-aasland commented Jul 28, 2022

Uh oh!

Uh oh!

erlend-aasland commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

erlend-aasland commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taleinat commented Aug 3, 2022

Uh oh!

erlend-aasland commented Aug 3, 2022

Uh oh!

ambv commented Aug 3, 2022

Uh oh!

erlend-aasland commented Aug 3, 2022

Uh oh!

terryjreedy left a comment

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

erlend-aasland Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 3, 2022

Uh oh!

ambv commented Aug 4, 2022

Uh oh!

Uh oh!

taleinat commented Oct 14, 2020 •

edited by terryjreedy

Loading

E-Paine commented Oct 14, 2020 •

edited

Loading

erlend-aasland commented Aug 3, 2022 •

edited

Loading

erlend-aasland commented Aug 3, 2022 •

edited

Loading