@njsmith Aside from the TODO's in the PR itself, this is a pretty complete implementation of the current version of PEP 558 now.

One other known issue: the header file updates still need to be modified to conform with the new conventions.

I don't know if this is already addressed elsewhere, but I'm getting a strange error. Running the below script with python.exe built with this branch (I removed an unused parameter to make it compile):

def foo():
    import pdb; pdb.set_trace()
    x=1

foo()

gives

$ ./python.exe jump.py
> /Users/henry/pep558/cpython/jump.py(3)foo()
-> x=1
(Pdb) x=123
(Pdb) c
python.exe(69485,0x10961d5c0) malloc: *** error for object 0x1029ee790: pointer being freed was not allocated
python.exe(69485,0x10961d5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

but this error is absent for certain values of x, say:

$ ./python.exe jump.py
> /Users/henry/pep558/cpython/jump.py(3)foo()
-> x=1
(Pdb) x=1234
(Pdb) c
$

Hmm, looks like I may have a refcounting bug, and the small integer cache is able to provoke it into crashing.

I can report that not all small numbers trigger this error. On my machine x = 11, say, is fine. Most perplexing!

Running test_frame consistently triggers a crash in test_clear_locals for me, so I'm focusing on that initially.

Last push resolves the test_frame segfault, but adds a TODO noting that the locals proxy may still misbehave after a frame has been cleared (or is in the process of being destroyed).

There's also a threading test which checks reference cycles aren't being created through the frames, which is currently failing (quite possibly for a related reason)

Ugh, the refcyle issue detected by test_threading is kinda inherent in the current implementation: once frame.f_locals gets created, there's now a cyclic reference between the frame and its own locals proxy.

I'm going to try forcibly breaking the cycle when an optimized frame exits (before the eval loop drops its own reference).

ncoghlan@7078632 fixed the test_sys_settrace crash (mismanaged refcounts when local variables were updated via the write-through proxy).

There's still another segfault in test_log_destroyed_pending_task in test_asyncio, though.

It appears test_asyncio was hitting the same bug as test_frame (the pop/delete support in the write-through proxy wasn't quite right), as they're both passing for me after the latest change.

This PR is now a solid proof of concept for the PEP, but there's a lot of cleanup to be done before it would be ready for merging (and that's assuming the PEP is accepted in its current form - the latest draft is still to be discussed on python-dev).

I've made notes in the PR description of all the missing pieces that I'm already aware of (including the Mac OS X build failure, which is a clang vs gcc issue).

Latest round of PEP review resulted in several changes to the proposed C API, including a clearer separation between additions to the stable ABI and additions to the full CPython C API. I created #18052 as a preparatory refactoring that migrates frameobject.h to the modern header layout that clearly separates the two APIs.

I made enough progress on migrating to the redesigned API described in https://discuss.python.org/t/pep-558-defined-semantics-for-locals/2936/11?u=ncoghlan that next weekend I'll update the PEP to cover this version of the API (I didn't want to do that until I'd made the initial draft updates here).

Codecov Report

Merging #3640 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3640   +/-   ##
=======================================
  Coverage   82.03%   82.03%           
=======================================
  Files        1956     1956           
  Lines      584511   584511           
  Branches    44467    44467           
=======================================
  Hits       479520   479520           
  Misses      95364    95364           
  Partials     9627     9627           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 161ad47...161ad47. Read the comment docs.

Thanks to a suggestion by @markshannon, I was able to get rid of the circular reference between the frame and the fast locals proxy objects (the frame has gone back to holding only the dynamic snapshot that it stores now, and the proxies are created on demand as part of the "frame.f_locals" get descriptor)