ctypes array cannot be returned by value on ARM64 #110190

pdegroote · 2023-10-01T19:29:34Z

Bug report

Bug description:

Dear,

I'm wrapping a C-library with the ctypes module on Darwin (ARM64). I have a function that simply initializes a C-struct and returns the struct by value. The C-struct contains one simple C-array type. When using ArrayTypes in Python, it seems the struct is not returned properly.

On Windows and Linux platforms the same code does work as expected.

The strange thing is that if I lay out the struct explicitly with singular types in Python (as I expect the layout of the struct is the same), it does work as expected on all tried platforms.

To reproduce, I first create a simple .c file with the struct and the initializing function. In Python, I give code that I use to compile (with gcc), and then three different ways to wrap the struct and function in Python using ctypes. I believe the three ways should be equivalent (and they are on Windows and Linux), but not on Darwin (ARM64).

I saved this file as ctypesexample.c

struct example{double coords[3];};
struct example set_defaults(){
    struct example ex;
    ex.coords[0] = 1.0;
    ex.coords[1] = 2.0;
    ex.coords[2] = 3.0;
    return ex;}

I saved this file as reproduce.py

import os
import subprocess
import ctypes

def compile():
    """Compile the c file using gcc."""
    subprocess.call(["gcc","-c","-Wall","-Werror","-fpic","ctypesexample.c"])
    subprocess.call(["gcc","-shared","-o","libctypesexample","ctypesexample.o"])
    assert os.path.isfile('libctypesexample')

# Three ways to wrap the struct:
class Example1(ctypes.Structure):
    _fields_ = [
            ('coords', ctypes.c_double * 3)
               ]

class CoordType(ctypes.Array):
    _type_ = ctypes.c_double
    _length_ = 3

class Example2(ctypes.Structure):
    _fields_ = [('coords', CoordType)]
    
class Example3(ctypes.Structure):
    _fields_ = [
            ('x', ctypes.c_double),
            ('y', ctypes.c_double),
            ('z', ctypes.c_double)
               ]

def run():
    # Load the shared library
    so_location = os.path.abspath("libctypesexample")
    _lib = ctypes.cdll.LoadLibrary(so_location)
    _lib.set_defaults.restype = Example1
    o = _lib.set_defaults()
    print(f"Example1: ({o.coords[0]}, {o.coords[1]}, {o.coords[2]})")
    
    _lib.set_defaults.restype = Example2
    o = _lib.set_defaults()
    print(f"Example2: ({o.coords[0]}, {o.coords[1]}, {o.coords[2]})")

    _lib.set_defaults.restype = Example3
    o = _lib.set_defaults()
    print(f"Example3: ({o.x}, {o.y}, {o.z})")

if __name__ == "__main__":
    if not os.path.isfile('libctypesexample'):
        compile()
    run()

I tried this code on Linux (debian) -- both in Docker and WSL. And this works perfectly. Equivalent code in Windows is also working fine. In the past, I have run similar code as above on many different versions of Python (3.5-3.11) on both platforms for years (and still) without problems. For example on WSL with the following platform info

uname_result(system='Linux', release='5.15.90.1-microsoft-standard-WSL2', version='#1 SMP Fri Jan 27 02:56:13 UTC 2023', machine='x86_64')

and Python:

Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux

I get this correct output:

Example1: (1.0, 2.0, 3.0)
Example2: (1.0, 2.0, 3.0)
Example3: (1.0, 2.0, 3.0)

While on Darwin (ARM) with the following platform info:

system='Darwin', release='22.1.0', version='Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103', machine='arm64')

and Python

Python 3.11.4 (main, Jun 12 2023, 11:39:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin

I get the following output:

Example1: (3.025693809e-314, 3.0256938013e-314, 2.1481083817e-314)
Example2: (3.025693809e-314, 3.0256938013e-314, 2.1481083817e-314)
Example3: (1.0, 2.0, 3.0)

In summary, the struct doesn't seem to be properly returned as value using array types, though with a literal listing of the types, it seems there is no problem. So there is a workaround, but as we're using ctypes for automatic wrapping through reflection and the code works properly on Windows and Linux, it would be quite some work to implement the workaround.

I hope my bug report finds you without any errors, and many thanks for giving the opportunity to report.

Best regards

CPython versions tested on:

3.10, 3.11

Operating systems tested on:

Linux, macOS, Windows

Linked PRs

The text was updated successfully, but these errors were encountered:

sunmy2019 · 2023-10-02T21:56:03Z

I looked into it. ctypes utilizes libffi to make foreign function calls, and ctypes generates wrong signatures.

Long words in short: stgdict->ffi_type_pointer is wrong. Current behavior would cause return types interpreted as ctypes.c_void_p * 3.

class Example1(ctypes.Structure):
    _fields_ = [("coords", ctypes.c_double * 3)]


class Example2(ctypes.Structure):
    _fields_ = [("coords", ctypes.c_void_p * 3)]


# Three ways to wrap the struct:
class Example3(ctypes.Structure):
    _fields_ = [("x", ctypes.c_void_p), ("y", ctypes.c_void_p), ("z", ctypes.c_void_p)]


def run():
    # Load the shared library
    so_location = os.path.abspath("libctypesexample")
    _lib = ctypes.cdll.LoadLibrary(so_location)

    _lib.set_defaults.restype = Example1
    o = _lib.set_defaults()
    print(
        f"Example1: ({float.hex(o.coords[0])}, {float.hex(o.coords[1])}, {float.hex(o.coords[2])})"
    )

    _lib.set_defaults.restype = Example2
    o = _lib.set_defaults()
    print(f"Example2: ({o.coords[0]:x}, {o.coords[1]:x}, {o.coords[2]})")

    _lib.set_defaults.restype = Example3
    o = _lib.set_defaults()
    print(f"Example3: ({o.x:x}, {o.y:x}, {o.z})")

Example1: (0x0.0ffffcc6b11d0p-1022, 0x0.0ffff95567ef8p-1022, 0x0.0p+0)
Example2: (ffffcc6b11d0, ffff95567ef8, None)
Example3: (ffffcc6b11d0, ffff95567ef8, None)

Notice the binary representation of Example1 has ffffcc6b11d0 which is same as Example2 and Example3.

sunmy2019 · 2023-10-02T23:43:25Z

The special handling is here:

cpython/Modules/_ctypes/stgdict.c

Lines 700 to 710 in 21a6263

    
           #define MAX_STRUCT_SIZE 16 
        
               if (arrays_seen && (size <= MAX_STRUCT_SIZE)) { 
        
                   /* 
        
                    * See bpo-22273. Arrays are normally treated as pointers, which is 
        
                    * fine when an array name is being passed as parameter, but not when 
        
                    * passing structures by value that contain arrays. On 64-bit Linux, 
        
                    * small structures passed by value are passed in registers, and in 
        
                    * order to do this, libffi needs to know the true type of the array 
        
                    * members of structs. Treating them as pointers breaks things. 
        
                    *

You seem to hit one special HVA calling convention of aarch64 which is currently not handled.
https://learn.microsoft.com/en-us/cpp/build/arm64-windows-abi-conventions?view=msvc-170#return-values

Types that are HFAs with four or fewer elements are returned in s0-s3, d0-d3, or v0-v3, as appropriate.

Details

Integral values are returned in x0.

Floating-point values are returned in s0, d0, or v0, as appropriate.

A type is considered to be an HFA or HVA if all of the following hold:

It's non-empty,
It doesn't have any non-trivial default or copy constructors, destructors, or assignment operators,
All of its members have the same HFA or HVA type, or are float, double, or neon types that match the other members' HFA or HVA types.

HVA values with four or fewer elements are returned in s0-s3, d0-d3, or v0-v3, as appropriate.

Types returned by value are handled differently depending on whether they have certain properties, and whether the function is a non-static member function. Types which have all of these properties,

they're aggregate by the C++14 standard definition, that is, they have no user-provided constructors, no private or protected - non-static data members, no base classes, and no virtual functions, and
they have a trivial copy-assignment operator, and
they have a trivial destructor,

and are returned by non-member functions or static member functions, use the following return style:

Types that are HFAs with four or fewer elements are returned in s0-s3, d0-d3, or v0-v3, as appropriate.
Types less than or equal to 8 bytes are returned in x0.
Types less than or equal to 16 bytes are returned in x0 and x1, with x0 containing the lower-order 8 bytes.
For other aggregate types, the caller shall reserve a block of memory of sufficient size and alignment to hold the result. The address of the memory block shall be passed as an additional argument to the function in x8. The callee may modify the result memory block at any point during the execution of the subroutine. The callee isn't required to preserve the value stored in x8.

All other types use this convention:

The caller shall reserve a block of memory of sufficient size and alignment to hold the result. The address of the memory block shall be passed as an additional argument to the function in x0, or x1 if $this is passed in x0. The callee may modify the result memory block at any point during the execution of the subroutine. The callee returns the address of the memory block in x0.

sunmy2019 · 2023-10-02T23:47:27Z

Now the cause is clear, ctypes did not handle one special rule of aarch64, but fixing it is beyond my ability.

diegorusso · 2023-11-29T13:44:02Z

Hello, I'm looking at this issue and working on a fix.

Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate.

diegorusso · 2023-12-01T18:22:17Z

Hello, I've investigated this further and actually this was happening across Arm platforms including Linux aarch64 and aarch32 and not just on Darwin. I've raised the PR that fixes the issue providing information and context. Feel free to review it and any feedback is welcome.

Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate.

…112604) Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate. (cherry picked from commit bc68f4a)

…2767) Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate. (cherry picked from commit bc68f4a)

…2766) Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate. (cherry picked from commit bc68f4a)

ambv · 2023-12-06T16:05:49Z

This is now fixed in (to be released) 3.11.8, 3.12.1, and 3.13.0 alpha 3.
Thanks, Diego! ✨ 🍰 ✨

ambv · 2023-12-06T20:22:05Z

While this is fixed, it uncovered that a similar failure is happening on PPC64LE. The new tests introduced by Diego here fail on our PPC64LE buildbots as well. See for instance:
https://buildbot.python.org/all/#/builders/559/builds/4236/steps/6/logs/stdio

on PPC64LE

…LE (#112818)

on PPC64LE (python#112818) (cherry picked from commit 9f67042)

…ngh-112604 on PPC64LE (pythonGH-112818) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

…n PPC64LE (GH-112818) (#112829) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

…ngh-112604 on PPC64LE (pythonGH-112818) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

zooba · 2024-01-26T16:36:55Z

I'm seeing the same test fail on Windows ARM64, though not quite in the same place:

======================================================================
FAIL: test_array_in_struct (test.test_ctypes.test_structures.StructureTestCase.test_array_in_struct) [Test3C]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_ctypes\test_structures.py", line 670, in test_array_in_struct
    self.assertEqual(result, expected, sut.cls)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 2.0 != 6.0

(I added with self.subTest(sut.cls.__name__) inside the loop to see that it's Test3C failing, so it's actually line 669 in main branch)

My build was using MSVC 19.37, which is known to have at least one ARM64 regression, so this may be another one. I haven't yet figured out how to confirm, so leaving some details here in case someone else knows before I get back to it.

diegorusso · 2024-01-26T18:02:02Z

@zooba thanks for reporting it. I'll try to have a look at this next week.

diegorusso · 2024-01-30T12:11:12Z

@zooba I have the fix, PR is coming.

Fix the same issue of PR python#112959 on Windows ARM64 platform when using MSVC compiler. MSVC predifined platform preprocessor macro is _M_ARM64.

Fix the same issue of PR python#112959 on Windows ARM64 platform when using MSVC compiler. MSVC predefined platform preprocessor macro is _M_ARM64.

…onGH-114753) (cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

zooba · 2024-01-30T23:55:46Z

Thanks for the quick fix for Windows! Sorry for hijacking the issue - I'll leave it open for the macOS issue.

(cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

diegorusso · 2024-01-31T14:03:54Z

The macOS issue has been fixed since the first PR but then the tests refactoring exposed the bug on different platforms. I mean we can keep this issue open until we cover all different platforms but I guess at some point we need to draw a line?
Should we rename/repurpose the issue for multi platforms? tier-1 and tier-2 platforms should be covered with the changes made so far.
I'm OK with whatever people think it is appropriate.

zooba · 2024-01-31T15:22:11Z

Let's just label it "ARM64" instead of Darwin, and if the only remaining issue is SPARC then we can probably close until someone has a machine handy to repro and test. I'll leave open for now in case someone already does, but if there's nothing heard, it can be closed.

Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms. This because on Arm platforms structs with at most 4 elements of any floating point type values can be passed through registers. If the type is double the maximum size of the struct is 32 bytes. On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate.

on PPC64LE (python#112818)

…12959) Fix the same issue of PR python#112604 on PPC64LE platform Refactor tests to make easier to add more platfroms if needed.

…onGH-114753)

kulikjak · 2024-02-13T12:31:50Z

Hi. I found some time to dig into this more and currently I suspect gcc to be the culprit - both Python and libffi seem to do what is expected. But it's definitely too soon to say for sure - I have asked those more knowledgeable internally to confirm that, so we'll see.

(It seems that gcc doesn't conform to ABI when a struct with small float/double arrays is returned from a function - when I change the test structs from one double array to several doubles, it works as expected).

I hope to find a fix soon, but I understand if you want to close this (although this seems to be a general SPARC issue, so Linux SPARC is likely affected as well). If so, can we perhaps skip those tests on SPARC?

kulikjak · 2024-04-24T14:56:27Z

Hi. The problem is resolved, the main issue is indeed in the GCC code generator, which is breaking the SPARC ABI.

Still, even with that fixed, similarly to Windows ARM64 (#114753), MAX_STRUCT_SIZE should be adjusted to 32 in stgdict. I created #118233 for that.

diegorusso · 2024-04-24T15:10:07Z

@kulikjak thanks for the update!

thesamesam · 2024-04-24T15:12:51Z

Is there a link to the GCC bug? Thanks.

kulikjak · 2024-04-25T08:39:59Z

Is there a link to the GCC bug? Thanks.

Sure, here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114416

pdegroote added the type-bug An unexpected behavior, bug, or error label Oct 1, 2023

sunmy2019 added the topic-ctypes label Oct 2, 2023

bedevere-app bot mentioned this issue Dec 1, 2023

gh-110190: Fix ctypes structs with array on Arm #112604

Merged

bedevere-app bot mentioned this issue Dec 5, 2023

[3.11] gh-110190: Fix ctypes structs with array on Arm (#112604) #112766

Merged

bedevere-app bot mentioned this issue Dec 5, 2023

[3.12] gh-110190: Fix ctypes structs with array on Arm (#112604) #112767

Merged

ambv closed this as completed Dec 6, 2023

ambv reopened this Dec 6, 2023

ambv added a commit to ambv/cpython that referenced this issue Dec 6, 2023

pythongh-110190: Temporarily skip new test introduced in pythongh-112604

9ab8719

on PPC64LE

bedevere-app bot mentioned this issue Dec 6, 2023

gh-110190: Temporarily skip new test introduced in gh-112604 on PPC64LE #112818

Merged

Yhg1s added a commit to ambv/cpython that referenced this issue Dec 7, 2023

Merge branch 'main' into pythongh-110190-skip-ppc64le

025d973

Yhg1s pushed a commit that referenced this issue Dec 7, 2023

gh-110190: Temporarily skip new test introduced in gh-112604 on PPC64…

9f67042

…LE (#112818)

ambv added a commit to ambv/cpython that referenced this issue Dec 7, 2023

pythongh-110190: Temporarily skip new test introduced in pythongh-112604

286ed01

on PPC64LE (python#112818) (cherry picked from commit 9f67042)

ambv added a commit to ambv/cpython that referenced this issue Dec 7, 2023

[3.12] pythongh-110190: Temporarily skip new test introduced in pytho…

951aa62

…ngh-112604 on PPC64LE (pythonGH-112818) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

bedevere-app bot mentioned this issue Dec 7, 2023

[3.12] gh-110190: Temporarily skip new test introduced in gh-112604 on PPC64LE (GH-112818) #112829

Merged

Yhg1s pushed a commit that referenced this issue Dec 7, 2023

[3.12] gh-110190: Temporarily skip new test introduced in gh-112604 o…

8f1c912

…n PPC64LE (GH-112818) (#112829) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

ambv added a commit to ambv/cpython that referenced this issue Dec 7, 2023

[3.11] pythongh-110190: Temporarily skip new test introduced in pytho…

832bb54

…ngh-112604 on PPC64LE (pythonGH-112818) (cherry picked from commit 9f67042) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

bedevere-app bot mentioned this issue Dec 7, 2023

[3.11] gh-110190: Temporarily skip new test introduced in gh-112604 on PPC64LE (GH-112818) #112830

Merged

zooba mentioned this issue Jan 26, 2024

Test failures on Windows ARM64 install #114272

Closed

bedevere-app bot mentioned this issue Jan 30, 2024

gh-110190: Fix ctypes structs with array on Windows ARM64 #114753

Merged

zooba pushed a commit that referenced this issue Jan 30, 2024

gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753)

a06b606

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 30, 2024

pythongh-110190: Fix ctypes structs with array on Windows ARM64 (pyth…

db70d31

…onGH-114753) (cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

bedevere-app bot mentioned this issue Jan 30, 2024

[3.11] gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753) #114774

Merged

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 30, 2024

pythongh-110190: Fix ctypes structs with array on Windows ARM64 (pyth…

08465e6

…onGH-114753) (cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

bedevere-app bot mentioned this issue Jan 30, 2024

[3.12] gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753) #114775

Merged

zooba pushed a commit that referenced this issue Jan 31, 2024

gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753)

d326d52

(cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

zooba pushed a commit that referenced this issue Jan 31, 2024

gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753)

4defc50

(cherry picked from commit a06b606) Co-authored-by: Diego Russo <diego.russo@arm.com>

zooba changed the title ~~ctypes array cannot be returned by value on darwin (ARM64)~~ ctypes array cannot be returned by value on ARM64 Jan 31, 2024

zooba added the pending The issue will be closed if no feedback is provided label Jan 31, 2024

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-110190: Temporarily skip new test introduced in pythongh-112604

d4c5624

on PPC64LE (python#112818)

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-110190: Fix ctypes structs with array on PPCLE64 (pythonGH-1…

b1df4a6

…12959) Fix the same issue of PR python#112604 on PPC64LE platform Refactor tests to make easier to add more platfroms if needed.

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-110190: Fix ctypes structs with array on Windows ARM64 (pyth…

1d201de

…onGH-114753)

kulikjak added a commit to kulikjak/cpython that referenced this issue Apr 24, 2024

pythongh-110190: Fix ctypes structs with array on SPARC

a125b13

bedevere-app bot mentioned this issue Apr 24, 2024

gh-110190: Fix ctypes structs with array on SPARC #118233

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ctypes array cannot be returned by value on ARM64 #110190

ctypes array cannot be returned by value on ARM64 #110190

pdegroote commented Oct 1, 2023 •

edited by bedevere-app bot

sunmy2019 commented Oct 2, 2023

sunmy2019 commented Oct 2, 2023 •

edited

sunmy2019 commented Oct 2, 2023

diegorusso commented Nov 29, 2023

diegorusso commented Dec 1, 2023

ambv commented Dec 6, 2023

ambv commented Dec 6, 2023

zooba commented Jan 26, 2024

diegorusso commented Jan 26, 2024

diegorusso commented Jan 30, 2024

zooba commented Jan 30, 2024

diegorusso commented Jan 31, 2024

zooba commented Jan 31, 2024

kulikjak commented Feb 13, 2024

kulikjak commented Apr 24, 2024

diegorusso commented Apr 24, 2024

thesamesam commented Apr 24, 2024

kulikjak commented Apr 25, 2024

ctypes array cannot be returned by value on ARM64 #110190

ctypes array cannot be returned by value on ARM64 #110190

Comments

pdegroote commented Oct 1, 2023 • edited by bedevere-app bot

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

sunmy2019 commented Oct 2, 2023

sunmy2019 commented Oct 2, 2023 • edited

sunmy2019 commented Oct 2, 2023

diegorusso commented Nov 29, 2023

diegorusso commented Dec 1, 2023

ambv commented Dec 6, 2023

ambv commented Dec 6, 2023

zooba commented Jan 26, 2024

diegorusso commented Jan 26, 2024

diegorusso commented Jan 30, 2024

zooba commented Jan 30, 2024

diegorusso commented Jan 31, 2024

zooba commented Jan 31, 2024

kulikjak commented Feb 13, 2024

kulikjak commented Apr 24, 2024

diegorusso commented Apr 24, 2024

thesamesam commented Apr 24, 2024

kulikjak commented Apr 25, 2024

pdegroote commented Oct 1, 2023 •

edited by bedevere-app bot

sunmy2019 commented Oct 2, 2023 •

edited