Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-38392: PyObject_GC_Track() validates object in debug mode #16615

Merged
merged 2 commits into from Oct 7, 2019

Conversation

@vstinner
Copy link
Member

vstinner commented Oct 7, 2019

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object to ensure that all object attributes visited by tp_traverse()
are initialized.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.

https://bugs.python.org/issue38392

@vstinner

This comment has been minimized.

Copy link
Member Author

vstinner commented Oct 7, 2019

This change is backward incompatible and I'm not sure that it's a good idea, so I marked is a WIP. See https://bugs.python.org/issue38392 for the rationale an discussion.

Copy link
Member

serhiy-storchaka left a comment

LGTM. Please add a NEWS entry.

@vstinner

This comment has been minimized.

Copy link
Member Author

vstinner commented Oct 7, 2019

LGTM. Please add a NEWS entry.

Should I add an entry in What's New in Python 3.9?

@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Oct 7, 2019

This change does not change the documented behavior. So I think that an entry in NEWS would be enough.

If you used PyObject_GC_Track() properly, nothing will changed for you. If you used it improperly, you should use it properly at first place.

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
@vstinner vstinner force-pushed the vstinner:gc_track_validate branch from 6962577 to f587a86 Oct 7, 2019
@vstinner vstinner removed the DO-NOT-MERGE label Oct 7, 2019
@vstinner vstinner changed the title [WIP] bpo-38392: PyObject_GC_Track() validates object in debug mode bpo-38392: PyObject_GC_Track() validates object in debug mode Oct 7, 2019
@vstinner

This comment has been minimized.

Copy link
Member Author

vstinner commented Oct 7, 2019

I modified my PR: if a bug is detected, _PyObject_ASSERT_FAILED_MSG() now gets the object passed to PyObject_GC_Track(). This object is more likely to help the developer debugging an issue, rather than displaying the visited object which is invalid and may only be displayed as "" without any further information.

I reintroduced https://bugs.python.org/issue33803 bug with this patch:

diff --git a/Python/hamt.c b/Python/hamt.c
index 28b4e1ef6c..d7dd555d15 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -2478,8 +2478,6 @@ hamt_alloc(void)
     if (o == NULL) {
         return NULL;
     }
-    o->h_count = 0;
-    o->h_root = NULL;
     o->h_weakreflist = NULL;
     PyObject_GC_Track(o);
     return o;

The hamt bug is immediately detected. Example (with tracemalloc enabled):

$ ./python -X tracemalloc=10 -m test -v test_context
(...)
0:00:00 load avg: 0.92 [1/1] test_context
test_context_copy_1 (test.test_context.ContextTest) ...
Modules/gcmodule.c:1930: visit_validate: Assertion failed: PyObject_GC_Track() object is not valid
Memory block allocated at (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616
  (...)

object address  : 0x7feb39ce19b0
object refcount : 1
object type     : 0x7f5ea0
object type name: hamt
object repr     : <hamt object at 0x7feb39ce19b0>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007feb48083740 (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322 in test_context_copy_1
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616 in _callTestMethod
  (...)
Aborted (core dumped)
  • The bug is detected
  • The object type and it's representation are logged
  • The Python traceback where the bug occurred is logged
  • The Python traceback where the object has been allocated is logged (this feature needs tracemalloc)

Note, when tracemalloc is disabled (default), this reminder is logged:

Enable tracemalloc to get the memory block allocation traceback

Onyly track the object in the GC once it's fully initialized.
@vstinner vstinner merged commit 1b18455 into python:master Oct 7, 2019
4 checks passed
4 checks passed
Azure Pipelines PR #20191007.42 succeeded
Details
bedevere/issue-number Issue number 38392 found
Details
bedevere/news News entry found in Misc/NEWS.d
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@vstinner vstinner deleted the vstinner:gc_track_validate branch Oct 7, 2019
jacobneiltaylor added a commit to jacobneiltaylor/cpython that referenced this pull request Dec 5, 2019
…GH-16615)

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.