Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-40334: Rewrite test_c_parser to avoid memory leaks #19694

Merged
merged 3 commits into from Apr 24, 2020

Conversation

@lysnikolaou
Copy link
Contributor

lysnikolaou commented Apr 23, 2020

Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.

With this PR every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.

https://bugs.python.org/issue40334

@vstinner

This comment has been minimized.

Copy link
Member

vstinner commented Apr 23, 2020

Why do you remove tests? It may be worth it to explain it in the commit message.

Copy link
Member

gvanrossum left a comment

I will land once tests pass.

@lysnikolaou lysnikolaou force-pushed the lysnikolaou:remove-test-cparser branch from 5731030 to 5f04be3 Apr 23, 2020
@lysnikolaou

This comment has been minimized.

Copy link
Contributor Author

lysnikolaou commented Apr 23, 2020

Why do you remove tests? It may be worth it to explain it in the commit message.

Commit message updated. Thanks for the advice.

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 23, 2020

(Hmm, there's a plan to skip the tests rather than delete the file.)

@lysnikolaou

This comment has been minimized.

Copy link
Contributor Author

lysnikolaou commented Apr 23, 2020

(Hmm, there's a plan to skip the tests rather than delete the file.)

Or maybe even fix them..

@lysnikolaou lysnikolaou marked this pull request as draft Apr 23, 2020
Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.

With this PR every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.
@lysnikolaou lysnikolaou force-pushed the lysnikolaou:remove-test-cparser branch from 5f04be3 to 4c02d3b Apr 24, 2020
@lysnikolaou lysnikolaou changed the title bpo-40334: Remove test_c_parser from test_peg_generator bpo-40334: Rewrite test_c_parser to avoid memory leaks Apr 24, 2020
@lysnikolaou lysnikolaou requested review from gvanrossum and pablogsal Apr 24, 2020
@lysnikolaou lysnikolaou marked this pull request as ready for review Apr 24, 2020
@lysnikolaou

This comment has been minimized.

Copy link
Contributor Author

lysnikolaou commented Apr 24, 2020

PR updated! It now fixes the tests that were causing the memory leaks, instead of removing them. (I didn't open a new one, because we needed to preserve the PR number)

@bedevere-bot

This comment has been minimized.

Copy link

bedevere-bot commented Apr 24, 2020

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit fcd1ada 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@pablogsal

This comment has been minimized.

Copy link
Member

pablogsal commented Apr 24, 2020

Checking with the buildbots to confirm that there were no more leaks.

@lysnikolaou

This comment has been minimized.

Copy link
Contributor Author

lysnikolaou commented Apr 24, 2020

On RHEL 7 it's only test_threading that leaks now.

@pablogsal

This comment has been minimized.

Copy link
Member

pablogsal commented Apr 24, 2020

On RHEL 7 it's only test_threading that leaks now.

Yup, all the buildbot failures are unrelated to this PR (and the refleaks ones do not leak anymore because of this) 🎉

@pablogsal pablogsal merged commit 24ffe70 into python:master Apr 24, 2020
51 of 64 checks passed
51 of 64 checks passed
Windows (x86)
Details
Windows (x64)
Details
macOS
Details
Ubuntu
Details
buildbot/AMD64 RHEL7 Refleaks PR Build done.
Details
buildbot/AMD64 RHEL8 Refleaks PR Build done.
Details
buildbot/PPC64LE RHEL7 Refleaks PR Build done.
Details
buildbot/s390x Fedora Refleaks PR Build done.
Details
buildbot/s390x RHEL7 Refleaks PR Build done.
Details
buildbot/s390x RHEL8 Refleaks PR Build done.
Details
buildbot/x86 Gentoo Non-Debug with X PR Build done.
Details
buildbot/AMD64 Fedora Stable Refleaks PR Build started.
Details
buildbot/AMD64 Windows8.1 Refleaks PR Build started.
Details
buildbot/PPC64LE Fedora Stable LTO PR Build started.
Details
buildbot/PPC64LE Fedora Stable PR Build started.
Details
buildbot/PPC64LE Fedora Stable Refleaks PR Build started.
Details
buildbot/x86 Gentoo Refleaks PR Build started.
Details
Azure Pipelines PR #20200424.14 succeeded
Details
bedevere/issue-number Issue number 40334 found
Details
bedevere/news "skip news" label found
buildbot/AMD64 Arch Linux TraceRefs PR Build done.
Details
buildbot/AMD64 Debian PGO PR Build done.
Details
buildbot/AMD64 Debian root PR Build done.
Details
buildbot/AMD64 Fedora Stable Clang Installed PR Build done.
Details
buildbot/AMD64 Fedora Stable Clang PR Build done.
Details
buildbot/AMD64 Fedora Stable LTO + PGO PR Build done.
Details
buildbot/AMD64 Fedora Stable LTO PR Build done.
Details
buildbot/AMD64 Fedora Stable PR Build done.
Details
buildbot/AMD64 FreeBSD Shared PR Build done.
Details
buildbot/AMD64 RHEL7 LTO + PGO PR Build done.
Details
buildbot/AMD64 RHEL7 LTO PR Build done.
Details
buildbot/AMD64 RHEL7 PR Build done.
Details
buildbot/AMD64 RHEL8 LTO + PGO PR Build done.
Details
buildbot/AMD64 RHEL8 LTO PR Build done.
Details
buildbot/AMD64 RHEL8 PR Build done.
Details
buildbot/AMD64 Ubuntu Shared PR Build done.
Details
buildbot/AMD64 Windows10 PR Build done.
Details
buildbot/AMD64 Windows8.1 Non-Debug PR Build done.
Details
buildbot/PPC64 Fedora PR Build done.
Details
buildbot/PPC64LE Fedora Stable Clang Installed PR Build done.
Details
buildbot/PPC64LE Fedora Stable Clang PR Build done.
Details
buildbot/PPC64LE Fedora Stable LTO + PGO PR Build done.
Details
buildbot/PPC64LE RHEL7 LTO + PGO PR Build done.
Details
buildbot/PPC64LE RHEL7 LTO PR Build done.
Details
buildbot/PPC64LE RHEL7 PR Build done.
Details
buildbot/aarch64 Fedora Stable PR Build done.
Details
buildbot/aarch64 RHEL7 PR Build done.
Details
buildbot/aarch64 RHEL8 PR Build done.
Details
buildbot/s390x Debian PR Build done.
Details
buildbot/s390x Fedora Clang Installed PR Build done.
Details
buildbot/s390x Fedora Clang PR Build done.
Details
buildbot/s390x Fedora LTO + PGO PR Build done.
Details
buildbot/s390x Fedora LTO PR Build done.
Details
buildbot/s390x Fedora PR Build done.
Details
buildbot/s390x RHEL7 LTO + PGO PR Build done.
Details
buildbot/s390x RHEL7 LTO PR Build done.
Details
buildbot/s390x RHEL7 PR Build done.
Details
buildbot/s390x RHEL8 LTO + PGO PR Build done.
Details
buildbot/s390x RHEL8 LTO PR Build done.
Details
buildbot/s390x RHEL8 PR Build done.
Details
buildbot/s390x SLES PR Build done.
Details
buildbot/x86 Gentoo Installed with X PR Build done.
Details
buildbot/x86-64 macOS PR Build done.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@lysnikolaou lysnikolaou deleted the lysnikolaou:remove-test-cparser branch Apr 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.