Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM64 macOS variadic arguments not passed properly in ctypes #92892

Open
Rapptz opened this issue May 17, 2022 · 7 comments
Open

ARM64 macOS variadic arguments not passed properly in ctypes #92892

Rapptz opened this issue May 17, 2022 · 7 comments
Labels
docs expert-ctypes OS-mac type-bug

Comments

@Rapptz
Copy link

@Rapptz Rapptz commented May 17, 2022

Bug report

Using ctypes with variadic functions on ARM64 macOS machines seems to improperly pass the arguments, leading to truncation.

Minimal repro:

>>> import ctypes
>>> from ctypes import util
>>> libc = ctypes.CDLL(util.find_library("c"))
>>> libc.printf(b"hello %d world\n", 128_000_000)
hello 0 world
14

This happens regardless of casting it to a ctypes.c_int explicitly or not.

>>> libc.printf(b"hello %ld world\n", ctypes.c_int(1))
hello 0 world
14

On my regular machine (in this case an x64 Windows machine) it works as expected:

>>> import ctypes
>>> libc = ctypes.cdll.msvcrt
>>> libc.printf(b"hello %d world\n", 128_000_000)
hello 128000000 world
22

Your environment

I do not personally have a macOS machine, but I got a few others who did have a machine test for me. Their versions were as follows:

Machine 1:
Python 3.10.1, macOS 12.3.1 (21E258)

Machine 2:

Python 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:37:23) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
macOS-12.2.1-arm64-arm-64bit

Machine 3:

~ % python3 --version
Python 3.9.10

~ % sw_vers
ProductName:     macOS
ProductVersion:  12.3.1
BuildVersion:    21E258
  • CPython versions tested on: 3.9, 3.10
  • Operating system and architecture: ARM64 Apple Silicon macOS
@ronaldoussoren
Copy link
Contributor

@ronaldoussoren ronaldoussoren commented May 18, 2022

For variadic functions on macOS/arm64 you must explicitly annotate the fixed arguments to the variadic function (for printf the format string argument).

The reason for that is that the macOS/arm64 calling convention is different between plain and variadic functions (for the variadic arguments themselves) and types switches between the to based on the number of arguments passed to the function.

This probably needs some documentation.

P.S. Sorry about the vague language, I'll write something more coherent later.

@ronaldoussoren
Copy link
Contributor

@ronaldoussoren ronaldoussoren commented May 18, 2022

To expand on my previous comment, the following code works as expected:

import ctypes
from ctypes import util
libc = ctypes.CDLL(util.find_library("c"))
libc.printf.argtypes = [ctypes.c_char_p].   # <--- argument type spec
libc.printf(b"hello %d world\n", 128_000_000)

The marked line just before calling printf tells ctypes about the expected types and number of fixed arguments, and that way the ctypes implementation knows that the additional integer argument should be passed using the varargs convention which is different than the argument passing convention for normal functions.

@ronaldoussoren ronaldoussoren added the docs label May 18, 2022
@ronaldoussoren
Copy link
Contributor

@ronaldoussoren ronaldoussoren commented May 18, 2022

I'm not sure how to properly document this requirement, probably a new subsection in the "Calling Functions" section?

@m13253
Copy link

@m13253 m13253 commented May 18, 2022

Thanks for the explanation. With your information, I fixed the original downstream issue.

I agree that we need to document this requirement. This is both a portability issue and a security issue, because the wrong value is passed into the external function without being noticed or detected.

I would suggest that we “recommend every variadic function to get its arguments specified”, because specifying the arguments does no harm on other platforms.

@m13253
Copy link

@m13253 m13253 commented May 18, 2022

For anyone coming to this page in the future, here is the Apple official documentation:

https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#Update-Code-that-Passes-Arguments-to-Variadic-Functions

(Read the section named “Update Code that Passes Arguments to Variadic Functions”)

@Rapptz
Copy link
Author

@Rapptz Rapptz commented May 18, 2022

Thanks for the explanation and the response, it's greatly appreciated!

I'm not sure how to properly document this requirement, probably a new subsection in the "Calling Functions" section?

Maybe near the bottom of this section https://docs.python.org/3/library/ctypes.html#calling-functions-continued an admonition detailing what needs to be done for ARM64 could be written? Something like...

.. warning::

    When calling variadic functions (such as ``printf``) certain platforms, such as ARM64 for Apple platforms, require specifying ``argtypes`` for the non-variadic parameters. In the case of ``printf`` this would be the ``const char*`` initial parameter.

    .. code-block:: python3

        libc.printf.argtypes = [ctypes.c_char_p].   # <--- argument type spec
    
    Failure to do this will result in undesirable behavior. Since specifying ``argtypes`` doesn't inhibit portability, it's recommended to do so for all variadic functions.

I think this would give it enough attention.

@ronaldoussoren
Copy link
Contributor

@ronaldoussoren ronaldoussoren commented May 18, 2022

Something like that, although I'd prefer a new heading about calling variadic functions instead of inserting a warning about macOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs expert-ctypes OS-mac type-bug
Projects
None yet
Development

No branches or pull requests

4 participants