Cython - Indexing numpy array within nogil function

Question

I'm trying to extend the Splitter class in sklearn, which works with sklearn's decision tree classes. More specifically, I want to add a feature_weights variable in the new class, which will affect the determination of the best split point by altering the purity calculations proportionally with the feature weights.

The new class is almost an exact copy of sklearn's BestSplitter class (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_splitter.pyx) with only minor changes. Here's what I have so far:

cdef class WeightedBestSplitter(WeightedBaseDenseSplitter):

    cdef object feature_weights # new variable - 1D array of feature weights

    def __reduce__(self):
        # same as sklearn BestSplitter (basically)

    # NEW METHOD
    def set_weights(self, object feature_weights): 
        feature_weights = np.asfortranarray(feature_weights, dtype=DTYPE)
        self.feature_weights = feature_weights  

    cdef int node_split(self, double impurity, SplitRecord* split,
                        SIZE_t* n_constant_features) nogil except -1:

        # .... same as sklearn BestSplitter ....

        current_proxy_improvement = self.criterion.proxy_impurity_improvement()
        current_proxy_improvement *= self.feature_weights[<int>(current.feature)]  # new line

        # .... same as sklearn BestSplitter ....

A couple notes about the above: I'm using the object variable type and np.asfortranarray because that is how the variable X is defined and set in other places and X is indexed like I'm trying to index feature_weights. Also, custom.feature has a variable type of SIZE_t per the _splitter.pxd file (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_splitter.pxd).

The issue seems to be created by the variable type of self.feature_weights. The above code throws multiple errors, but even trying to reference something like self.feature_weights[0] and set it to another variable throws the error:

Indexing Python object not allowed without gil

I'm wondering what I need to do to be able to index self.feature_weights and use the scalar value as a multiplier.

Searching on cython and nogil, produces (among other things): lbolla.info/python-threads-cython-gil, "Cython offers a wonderful context manager to run instructions without the GIL: with nogil. The catch is that it can only be used to run code that does not touch any Python object." — hpaulj
– hpaulj, Commented Jun 2, 2020 at 4:38

DavidW · Accepted Answer · 2020-06-02 05:48:56Z

3

You definitely can't index a generic Python object without the GIL (as you're trying to do). You can index typed memoyviews without the GIL.

Define feature_weights as

cdef double[:] feature_weights

answered Jun 2, 2020 at 5:48

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SuperCodeBrah Over a year ago

This does the trick. I'm not used to casting in that way, if that's even the right term. Any idea why self.X can be of type object and still be indexed? I traced it back to X = np.asfortranarray(X, dtype=DTYPE) within the TreeBuilder class (github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/…), similar to the way I have it above.

DavidW Over a year ago

It's just a Numpy array. It defines __getitem__ so it can be indexed in Python. Defining as a typed memoryview gives Cython a direct way of accessing the data so the indexing works faster and doesn't need the GIL.

Collectives™ on Stack Overflow

Cython - Indexing numpy array within nogil function

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related