Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-89727: Fix pathlib.Path.walk RecursionError on deep trees #100282

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

Ovsyanka83
Copy link
Contributor

@Ovsyanka83 Ovsyanka83 commented Dec 15, 2022

Use a stack to implement pathlib.Path.walk iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees.

@netlify
Copy link

netlify bot commented Dec 15, 2022

Deploy Preview for python-cpython-preview canceled.

Name Link
🔨 Latest commit 2bfc47d
🔍 Latest deploy log https://app.netlify.com/sites/python-cpython-preview/deploys/639c50140be628000884140d

@Ovsyanka83
Copy link
Contributor Author

Ovsyanka83 commented Dec 15, 2022

TODO:

  • Add a news blip
  • Add a test that checks for deeply nested directories
  • Figure out a better name for is_result

Update:
All done!

@Ovsyanka83 Ovsyanka83 marked this pull request as ready for review Dec 16, 2022
@Ovsyanka83 Ovsyanka83 requested a review from brettcannon as a code owner Dec 16, 2022
@pitrou pitrou requested a review from serhiy-storchaka Dec 16, 2022
@brettcannon
Copy link
Member

brettcannon commented Dec 16, 2022

/cc @barneygale

Lib/pathlib.py Outdated Show resolved Hide resolved
path = pathlib.Path(base, *(['d']*50))
path.mkdir(parents=True)

with infinite_recursion(40):
Copy link
Contributor

@carljm carljm Dec 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that it was merged in the os.walk PR, maybe use test.support.set_recursion_limit here? Although in practice the functionality is the same, the name infinite_recursion is inappropriate to this use and may confuse the reader regarding the purpose of the test (this test has nothing to do with infinite recursion.)

Suggested change
with infinite_recursion(40):
with set_recursion_limit(40):

Copy link
Contributor Author

@Ovsyanka83 Ovsyanka83 Dec 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! That was my intention all along to also use set_recursion_limit but I wanted to wait until it gets merged to prevent duplicate commits and possible merge conflicts that could arise if I did anything slightly differently.

carljm
carljm approved these changes Dec 20, 2022
Copy link
Contributor

@carljm carljm left a comment

LGTM. I assume because of the initial underscore we don't have to worry about people who may have subclassed Path and overridden the _walk method (now to no effect.)

@Ovsyanka83
Copy link
Contributor Author

Ovsyanka83 commented Dec 20, 2022

@carljm remember that Path.walk has only been created in 3.12 which means that 99% of libraries/projects do not and cannot depend on it yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants