Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite recursion in Pickler.persistent_id #89850

Open
embe mannequin opened this issue Nov 2, 2021 · 3 comments
Open

Infinite recursion in Pickler.persistent_id #89850

embe mannequin opened this issue Nov 2, 2021 · 3 comments
Assignees
@serhiy-storchaka
Labels
3.10 3.11 3.12 interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@embe
Copy link
Mannequin

embe mannequin commented Nov 2, 2021 •
edited by bedevere-bot

BPO 45687
Nosy @serhiy-storchaka, @embe
Files
  • pickle-bug.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2021-11-02.08:21:56.849>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11', 'library']
title = 'Infinite recursion in Pickler.persistent_id'
updated_at = <Date 2021-11-02.09:16:16.272>
user = 'https://github.com/embe'

    bugs.python.org fields:

    activity = <Date 2021-11-02.09:16:16.272>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2021-11-02.08:21:56.849>
creator = 'embe-navalgo'
dependencies = []
files = ['50418']
hgrepos = []
issue_num = 45687
keywords = []
message_count = 1.0
messages = ['405494']
nosy_count = 2.0
nosy_names = ['serhiy.storchaka', 'embe-navalgo']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45687'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Linked PRs
    @embe
    Copy link
    Mannequin Author

    embe mannequin commented Nov 2, 2021 •
    edited by iritkatriel

    The following code, which seems reasonable:

    import io
import pickle

class Pickler(pickle.Pickler):
  def persistent_id(self, obj):
    return super().persistent_id(obj)

Pickler(io.BytesIO()).dump(42)

    crashes with:
    RecursionError: maximum recursion depth exceeded while calling a Python object

    It works perfectly when inheriting from pickle._Pickler (the Python implementation).

    @embe embe mannequin added 3.7 3.8 3.10 3.9 stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Nov 2, 2021
    @serhiy-storchaka serhiy-storchaka added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.11 and removed 3.7 3.8 labels Nov 2, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @iritkatriel
    Copy link
    Member

    iritkatriel commented Jun 1, 2022

    I think it's because in the C version persistent_id can be NULL (but it's still in dir()):

    >>> pickle.Pickler(io.BytesIO()).persistent_id
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: persistent_id
>>> dir(pickle.Pickler(io.BytesIO()))
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'bin', 'clear_memo', 'dispatch_table', 'dump', 'fast', 'memo', 'persistent_id']
>>>

    on the other hand:

    >>> pickle._Pickler(io.BytesIO()).persistent_id
<bound method _Pickler.persistent_id of <pickle._Pickler object at 0x10ce42220>>

    @iritkatriel iritkatriel added 3.12 and removed 3.9 labels Oct 7, 2022
    @bedevere-bot bedevere-bot mentioned this issue Dec 19, 2022
    Closed
    @furkanonder
    Copy link
    Sponsor Contributor

    furkanonder commented Dec 20, 2022 

    (gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 3
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$1 = void
(gdb) c
Continuing.

Breakpoint 1, Pickler_get_persid (self=0x7ffff75c13e0, _unused_ignored=0x0) at ./Modules/_pickle.c:5004
5004        if (self->pers_func == NULL) {
(gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 4
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$2 = void
(gdb) c
Continuing.

Breakpoint 1, Pickler_get_persid (self=0x7ffff75c13e0, _unused_ignored=0x0) at ./Modules/_pickle.c:5004
5004        if (self->pers_func == NULL) {
(gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 5
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$3 = void
(gdb)

    The refcount of self->pers_func appears to be increasing which causes the recursion error. I tried to decrease the refcount (I reverted the changes for this function on this PR, by the way, I didn't see any reference cycles @serhiy-storchaka ) Recursion error is fixed but I am getting TypeError

    Traceback (most recent call last):
  File "/home/furkan/cpython/bug.py", line 8, in <module>
    Pickler(io.BytesIO()).dump(42)
  File "/home/furkan/cpython/bug.py", line 6, in persistent_id
    return super().persistent_id(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Pickler.persistent_id() missing 1 required positional argument: 'obj'

    @serhiy-storchaka serhiy-storchaka self-assigned this Dec 22, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @serhiy-storchaka serhiy-storchaka

    Labels
    3.10 3.11 3.12 interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @iritkatriel @serhiy-storchaka @furkanonder