Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-45256: Avoid C calls for more Python to Python calls. #28937

Merged
merged 6 commits into from Oct 18, 2021

Conversation

@markshannon
Copy link
Contributor

@markshannon markshannon commented Oct 13, 2021

Extends the approach used for CALL_FUNCTION to CALL_FUNCTION_KW, CALL_METHOD and CALL_METHOD_KW.

Also modifies initialize_locals and _PyTuple_FromArraySteal to have the same behavior w.r.t. reference counting regardless of whether they succeed or fail.

https://bugs.python.org/issue45256

@Fidget-Spinner
Copy link
Contributor

@Fidget-Spinner Fidget-Spinner commented Oct 13, 2021

Did anyone discover what caused the slight slowdown in the initial PR's benchmark?

…onsume the argument references regardless of whether they succeed or fail.
@markshannon
Copy link
Contributor Author

@markshannon markshannon commented Oct 14, 2021

Did anyone discover what caused the slight slowdown in the initial PR's benchmark?

Maybe the additional tests for consuming references?
I don't think it matters much, as specialization will handle all the fast paths.

@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 14, 2021

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 7c0f498 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@markshannon
Copy link
Contributor Author

@markshannon markshannon commented Oct 14, 2021

@markshannon markshannon requested a review from pablogsal Oct 14, 2021
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 14, 2021

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 03e7ad9 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@@ -5480,7 +5459,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con,
_PyErr_Format(tstate, PyExc_TypeError,
"%U() keywords must be strings",
con->fc_qualname);
goto fail;
goto kw_fail;
Copy link
Member

@pablogsal pablogsal Oct 14, 2021

I find this a bit confusing. How this is this cleaning the positional arguments in the case where some positional args have been copied and we fail to copy some of the keywords?

Copy link
Contributor Author

@markshannon markshannon Oct 14, 2021

All the positional arguments have been copied at this point.

Copy link
Member

@pablogsal pablogsal Oct 14, 2021

Yeah exactly, so if the call fails you need to increment those references so the cleanup succeeds, no? This is because if the references have been stolen, we own the references back to the positional arguments

Copy link
Contributor Author

@markshannon markshannon Oct 14, 2021

If steal_args is true, then we always consume the references. #28937 (comment)

By this point the references to all positional arguments have been consumed, as have all references of kwargs up to i (exclusive).
kw_fail consumes the references to kwargs from i (inclusive) to kwcount (exclusive) so it can them jump to fail_late as all references will have been consumed.

Copy link
Member

@pablogsal pablogsal Oct 14, 2021

Ah, wait a minute, I misread what we are doing! We are consuming references now, while before we were increasing them so the stack cleans them.

Copy link
Contributor Author

@markshannon markshannon Oct 14, 2021

👍

Copy link
Member

@pablogsal pablogsal Oct 14, 2021

Ah, we are avoiding the double free by shrinking the stack here, no?

                  STACK_SHRINK(stackadj);
                    // The frame has stolen all the arguments from the stack,
                    // so there is no need to clean them up.
                    Py_XDECREF(kwnames);
                    Py_DECREF(function);

Copy link
Contributor Author

@markshannon markshannon Oct 15, 2021

👍

@@ -1702,6 +1702,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr
switch (opcode) {
#endif

/* Variables used for making calls */
PyObject *kwnames;
Copy link
Member

@pablogsal pablogsal Oct 14, 2021

Could we put this into a struct instead of having more locals? It makes it a bit more clean to read and contextualize and it will have the same performance as long as the struct is stack allocated

Copy link
Contributor Author

@markshannon markshannon Oct 14, 2021

Do we want to rely on all compilers being able to do perfect escape analysis here?
I'd rather leave these as local variables so the compiler can easily allocate them to registers?

Copy link
Member

@pablogsal pablogsal Oct 14, 2021

There is no way that given the size of this function this is going to end in registers. Indeed: I checked with GCC, clang, ICC and xlc in all different optimization level and not a single one places these locals on registers.

Up to you anyway, I don't feel super strongly about it but I think it helps with clarity and organization.

Copy link
Contributor Author

@markshannon markshannon Oct 15, 2021

How did you determine which of these were in registers? Once in SSA form, there are 15 (I think) variables here.

Copy link
Member

@pablogsal pablogsal Oct 15, 2021

gdb/dbx + breakpoint for this function + disas + info registers. I checked if any of the values in these locals are stored or partially stored in registers.

I only checked x86-64 thought.

Python/ceval.c Outdated Show resolved Hide resolved
Python/ceval.c Show resolved Hide resolved
Python/ceval.c Show resolved Hide resolved
Python/ceval.c Outdated Show resolved Hide resolved
Copy link
Member

@pablogsal pablogsal left a comment

I left some minor comments, otherwise LGTM.

Great work! I have not checked for refleaks, will do this afterwards but I also scheduled a buildbot run

@markshannon
Copy link
Contributor Author

@markshannon markshannon commented Oct 14, 2021

I ran the buildbots earlier. All good except the Gentoo ones failing for tkinter stuff.
Feel free to re-run them, in case I broke something in the last couple of commits.

@markshannon markshannon merged commit 70945d5 into python:main Oct 18, 2021
11 checks passed
@markshannon markshannon deleted the flatten-all-py-calls branch Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants