While considering the situation of gh-94215, I realized there is a broader concern worth discussing, of which that issue is a specific instance. FWIW, this isn't some massive, pervasive, nor high-priority problem that needs immediate attention. It is worth consideration at some point, though, as it's bitten us already and I expect it will bite us again.

Context:

In the eval loop (in ceval.c), we copy some interpreter/thread/frame state into local variables, to reduce overhead a little. Of these variables, several get modified during execution. One of them, stack_pointer, must be kept in sync with the frame->stacktop, which can be modified via _PyFrame_SetStackPointer(), _PyFrame_StackPop(), and _PyFrame_StackPush(). We pass stack_pointer in various calls (e.g. vectorcall) throughout the eval loop, so it is important that it stay in sync, at least at the points where we use it. (I covered this a bit more in the other issue.)

The Concern:

From where I'm at, it's hard to have a lot of confidence that we are preserving the invaraiant (that stack_pointer is properly in sync with the frame state) in all cases. There doesn't seem to be any structural mechanism by which we get guarantees (relative to use of _PyFrame_SetStackPointer(), etc.). Instead it seems like knowing when to synchronize stack_pointer, and when we can avoid unnecessary synchronization, currently requires deep knowledge of a lot of pieces.

Consequently, it seems like it is relatively easy to break the invariant accidentally. The fact that it happened to someone as knowledgeable as @markshannon really says a lot.

Am I out of touch here? If not, is it worth doing something about all that?

CC @iritkatriel @tiran @markshannon @pablogsal