Fast string array - Cython

Question

Having following hypothetical code:

cdef extern from "string.h":
    int strcmp(char* str1, char* str2)

def foo(list_str1, list_str2):
    cdef unsigned int i, j
    c_arr1 = ??
    c_arr2 = ??
    for i in xrange(len(list_str1)):
        for j in xrange(len(list_str2)):
            if not strcmp(c_arr1[i], c_arr2[j]):
                do some funny stuff

is there some way how to convert the lists to c arrays?

I have read and tried Cython - converting list of strings to char ** but that only throws errors.

Added recently Python 3 solution here of similar task, maybe you're interested in reading it. — Arty
– Arty, Commented Nov 14, 2021 at 12:49

falsetru · Accepted Answer · 2013-07-07 11:32:00Z

17

Try following code. to_cstring_array function in the following code is what you want.

from libc.stdlib cimport malloc, free
from libc.string cimport strcmp
from cpython.string cimport PyString_AsString

cdef char ** to_cstring_array(list_str):
    cdef char **ret = <char **>malloc(len(list_str) * sizeof(char *))
    for i in xrange(len(list_str)):
        ret[i] = PyString_AsString(list_str[i])
    return ret

def foo(list_str1, list_str2):
    cdef unsigned int i, j
    cdef char **c_arr1 = to_cstring_array(list_str1)
    cdef char **c_arr2 = to_cstring_array(list_str2)

    for i in xrange(len(list_str1)):
        for j in xrange(len(list_str2)):
            if i != j and strcmp(c_arr1[i], c_arr2[j]) == 0:
                print i, j, list_str1[i]
    free(c_arr1)
    free(c_arr2)

foo(['hello', 'python', 'world'], ['python', 'rules'])

edited Jul 7, 2013 at 11:32

answered Jul 7, 2013 at 11:23

falsetru

371k69 gold badges770 silver badges660 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ead Over a year ago

PyString_AsString is python2 only, so this solution will not work for python3

falsetru Over a year ago

@ead, Beside PyString_AsString, there are xrange calls in OP's code. So I thought it's okay to assume it's python 2 code. Any suggestion to make this solution work both in python 2/3 is welcome.

Celdor Over a year ago

I have found out that in Py3 PyUnicode_AsUTF8 is supposed to be used but I run into error: Storing unsafe C derivative of temporary Python reference. When I factorized the code assigning PyUnicode_AsUTF8(list_str[i]) to a temp variable, I run into another error: 'PyUnicode_AsUTF8' is not a constant, variable or function identifier. I have no clue how to proceed at this point.

DavidW Over a year ago

It's worth making clear that the memory for the stored char* is owned by the Python strings so is only valid while the Python strings are still alive. It's fine in this question, but I came here from a linked question that had copied the code and run into problems

ead · Accepted Answer · 2019-03-28 08:41:13Z

14

If you're on Python 3, here's an update to @falsetru's answer (untested on Python 2).

cdef extern from "Python.h":
    char* PyUnicode_AsUTF8(object unicode)

from libc.stdlib cimport malloc, free
from libc.string cimport strcmp

cdef char ** to_cstring_array(list_str):
    cdef char **ret = <char **>malloc(len(list_str) * sizeof(char *))
    for i in xrange(len(list_str)):
        ret[i] = PyUnicode_AsUTF8(list_str[i])
    return ret

def foo(list_str1, list_str2):
    cdef unsigned int i, j
    cdef char **c_arr1 = to_cstring_array(list_str1)
    cdef char **c_arr2 = to_cstring_array(list_str2)

    for i in range(len(list_str1)):
        for j in range(len(list_str2)):
            if i != j and strcmp(c_arr1[i], c_arr2[j]) == 0:
                print(i, j, list_str1[i])
    free(c_arr1)
    free(c_arr2)

foo(['hello', 'python', 'world'], ['python', 'rules'])

Warning: The pointer returned by PyUnicode_AsUTF8 is cached in the parent unicode-object. Which has two consequences:

this pointer is only valid as long as the parent unicode-object is alive. Accessing it afterwards leads to undefined behavior (e.g. possible segmentation fault).
The caller of the PyUnicode_AsUTF8 isn't responsible for the freeing the memory.

edited Mar 28, 2019 at 8:41

ead

34.8k8 gold badges101 silver badges171 bronze badges

answered Jan 7, 2019 at 20:07

user666

5,7212 gold badges29 silver badges35 bronze badges

2 Comments

ead Over a year ago

Maybe it is worth mentioning, that since Python 3.7 it is const char * PyUnicode_AsUTF8(...) docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUTF8

user666 Over a year ago

Thank you, please edit the answer if you feel it needs correction!

Collectives™ on Stack Overflow

Fast string array - Cython

2 Answers 2

4 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related