CTypes: Numpy Array Always has Different Values

Question

I'm trying to get an array from CTypes but it always has diffirent values. Here is my simple code:

C++:

struct OutArray {
    int     array_len;
    float*  array;
};

extern "C" // required when using C++ compiler
__declspec(dllexport) void getArray(OutArray *ret) {

    std::vector<float>  array_test;

    for (int i = 0; i < 100; i++) {
            array_test.push_back(0.3);
            array_test.push_back(0.1);
            array_test.push_back(0.2);
        }

    std::cout << array_test[0] << " C++ First Value "  << std::endl;

    // Return Array and its Length
    ret->array = array_test.data();
    ret->array_len = array_test.size();

    std::cout << array_test.data() << " C++ Array Object " << std::endl;

}

Python:

class OutArray(Structure):
    _fields_ = [("array_len", ctypes.c_int),
                ("array", ctypes.POINTER(ctypes.c_float))]

lib.getArray.argtypes = []


array_test = OutArray()

return_val = lib.getArray(ctypes.byref(array_test))

print('Py Return Value: ', return_val)
print(array_test.array_len)
new_array = np.ctypeslib.as_array(array_test.array, shape=(array_test.array_len,))

print('Py New Array: ', new_array)
print('Py New Array Type: ', new_array.dtype)
print('Py First Value * 2: ', n

And always when I run my code from python(in Blender 3d editor) I get different results. Sometimes they are correct but sometimes not. 50 to 50.

Log1:

0.3 C++ First Value
0000027C55F34B40 C++ Array Object
Py Return Value:  1
300
Py New Array:  [1.3229004e+03 8.9122582e-43 1.2467500e+03 8.9122582e-43 1.0000000e-01
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01
 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01
 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01
...........
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01
 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01
 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01
 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01 3.0000001e-01
 1.0000000e-01 2.0000000e-01 3.0000001e-01 1.0000000e-01 2.0000000e-01]
Py New Array Type:  float32
Py First Value * 2:  2645.80078125

Log2:

0.3 C++ First Value
0000022613667DA0 C++ Array Object
Py Return Value:  1
300
Py New Array:  [0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2
 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2]
Py New Array Type:  float32
Py First Value * 2:  0.0

You could see how I run the code 2 time and I get different results. Also, I noticed even if the array of Log2 is correct but Py First Value * 2 = 0. It should be 0.6!

I assume that could be a problem with Sync or Blender. Thank you!

Forget the Python side for now - your C++ is broken. It doesn't matter what Python you're using to access your broken C++. — user2357112
– user2357112, Commented Apr 16, 2021 at 8:11
You can't access a vector's data buffer after its lifetime has ended. — user2357112
– user2357112, Commented Apr 16, 2021 at 8:13
When your void getArray() finishes std::vector<float>::~vector() is called. And then control returns back to python. And whoops:-( — hetepeperfan
– hetepeperfan, Commented Apr 16, 2021 at 8:18
Yes return something that is still alive after your function returns. — hetepeperfan
– hetepeperfan, Commented Apr 16, 2021 at 8:19
__declspec(dllexport) ties your code to very specific platforms — hetepeperfan
– hetepeperfan, Commented Apr 16, 2021 at 8:22

CristiFati · Accepted Answer · 2021-04-16 15:42:37Z

Listing [Python.Docs]: ctypes - A foreign function library for Python.

There are several issues with your code (as mentioned in the comments).
The main one is that you rely on the existence of an object when it no longer exists. The object in question is the array_test vector, which goes out of scope at the end of the getArray function, and therefore it gets destroyed.

So what we've got here is Undefined Behavior (which explains the "funny" behavior you're seeing). As a note, it's very likely Python will also crash (with segfault or Access Violation).

In order to fix the problem, you have to dynamically allocate the data. As a consequence, in order to avoid memory leaks, you need to deallocate it as well (manually), so you need another function to do that.

Below, it's a working example - I changed the names, and also I changed the function that populates the array to (create and) return it, instead of modifying an existing one (got as an argument).

dll00.cpp:

#include <stdio.h>

#if defined(_WIN32)
#  define DLL00_EXPORT_API __declspec(dllexport)
#else
#  define DLL00_EXPORT_API
#endif


typedef struct FloatArray {
    int size = 0;
    float *data = nullptr;
} FloatArray;


#if defined(__cplusplus)
extern "C" {
#endif

DLL00_EXPORT_API FloatArray* create(int size);
DLL00_EXPORT_API void destroy(FloatArray *array);

#if defined(__cplusplus)
}
#endif


FloatArray* create(int size) {
    if (size <= 0)
        return nullptr;
    FloatArray *fa = new FloatArray();
        fa->size = size * 3;
        fa->data = new float[fa->size];
        for (int i = 0; i < fa->size; i += 3) {
            fa->data[i] = 0.3;
            fa->data[i + 1] = 0.2;
            fa->data[i + 2] = 0.1;
        }
    return fa;
}


void destroy(FloatArray *array) {
    delete[] array->data;
    array->size = 0;
    delete array;
}

code00.py:

#!/usr/bin/env python

import sys
import ctypes as ct
import numpy as np


DLL_NAME = "./arrays.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")


class FloatArray(ct.Structure):
    _fields_ = [
        ("size", ct.c_int),
        ("data", ct.POINTER(ct.c_float)),
    ]


FloatArrayPtr = ct.POINTER(FloatArray)


def main(*argv):
    arrays = ct.CDLL(DLL_NAME)
    create = arrays.create
    create.argtypes = (ct.c_int,)
    create.restype = FloatArrayPtr

    destroy = arrays.destroy
    destroy.argtypes = (FloatArrayPtr,)
    destroy.restype = None

    arr_ptr = create(100)
    if not arr_ptr:
        print("`create` returned invalid array")
        return -1
    arr = arr_ptr.contents
    display_elems_count = 10
    print("Len: {:}\nFirst ({:}) elements: ".format(arr.size, display_elems_count), end="")
    for i in range(display_elems_count):
        print("{:.3f}".format(arr.data[i]), end=" ")
    print("...\n")

    np_arr = np.ctypeslib.as_array(arr.data, shape=(arr.size,))
    print(np_arr.dtype, np_arr.shape)
    print("NP Array:", np_arr)
    print("Destroying...")
    destroy(arr_ptr)


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q067121461]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2017\VC\Auxiliary\Build\vcvarsall.bat" x64 >nul

[prompt]> dir /b
arrays.so
code00.py
dll00.cpp

[prompt]> cl /nologo /MD /DDLL dll00.cpp  /link /NOLOGO /DLL /OUT:arrays.dll
dll00.cpp
   Creating library arrays.lib and object arrays.exp

[prompt]> dir /b
arrays.dll
arrays.exp
arrays.lib
arrays.so
code00.py
dll00.cpp
dll00.obj

[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.08.07_test0\Scripts\python.exe" code00.py
Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] 64bit on win32

Len: 300
First (10) elements: 0.300 0.200 0.100 0.300 0.200 0.100 0.300 0.200 0.100 0.300 ...

float32 (300,)
NP Array: [0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1
 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1 0.3 0.2 0.1]
Destroying...

Done.

Thank you a lot! What if I don't know the array's length (100 in this case)? The array will be generated in the C++ part and returned to Python.
If you don't know the array length, then you simply ignore the argument in C++. I added it to be more generic (and customizable).

Collectives™ on Stack Overflow

CTypes: Numpy Array Always has Different Values

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related