Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-95865: Further reduce quote_from_bytes memory consumption #96860

Merged
merged 1 commit into from Sep 19, 2022

Conversation

gpshead
Copy link
Member

@gpshead gpshead commented Sep 16, 2022

Further reduce quote_from_bytes memory consumption on large input values. Based on Dennis Sweeney's chunking idea.

on large input values.  Based on Dennis Sweeney's chunking idea.
@gpshead gpshead requested a review from sweeneyde Sep 16, 2022
@gpshead gpshead added type-feature A feature request or enhancement performance Performance or resource usage stdlib Python modules in the Lib dir labels Sep 16, 2022
@gpshead
Copy link
Member Author

gpshead commented Sep 16, 2022

On large values, this consumes ~half the memory.

@sweeneyde
Copy link
Member

sweeneyde commented Sep 16, 2022

This looks like a correct, comprehensible change, my only wonder would be whether some sort of regex-based approach would be more efficient for this quote direction as well, i.e., have a regex recognize strings of safe characters.

@sweeneyde
Copy link
Member

sweeneyde commented Sep 16, 2022

It may also depend on what sorts of strings are typically quoted/unquoted. Different texts/corpuses probably have different performance characteristics, and I am not all that familiar with how this gets used in the wild.

@gpshead
Copy link
Member Author

gpshead commented Sep 19, 2022

there are likely further optimization opportunities. but as this does use notably less memory already, lets go ahead and do it. I don't like introducing regexes when possible that could be tested and considered as a separate change if anyone ever cares here.

@gpshead gpshead merged commit e61ca22 into python:main Sep 19, 2022
14 checks passed
@gpshead gpshead deleted the gh/95865/quote_mem_use branch Sep 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants