Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upMore Memory-efficient Implementation of itertools.cycle #17783
Conversation
Requires no additional copy of the iterable.
This comment has been minimized.
This comment has been minimized.
the-knights-who-say-ni
commented
Jan 1, 2020
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). Recognized GitHub usernameWe couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames: This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
youkaichao commentedJan 1, 2020
Requires no additional copy of the iterable.
Standard implementation of
cycle
looks like:We propose to change it to
Drawback: a little incompatible with the previous
cycle
v1 can accept an iteratable which can be iterated only once while v2 cannot.
Benefit: more memory-efficient
It doesn't require an additional copy.
I think v2 is better. If the caller wants to
cycle
through an iteratable (namedit
) which can be iterated only once, he can usecycle(list(it))
. In most cases, the caller holds repeatable iterable, socycle(it)
is ok without additional memory consumption.Benefit for DeepLearning
And there are cases where multiple iter calls to an iterable returns different iterators! The dataloader for DeepLearning is a perfect example:
What's more, in DeepLearning, it is not possible to store the whole dataset in memory, but the dataloader can be called multiple times to get the iterator:
So I hope we can get this more memory efficient
cycle
:)