Converting pandas dataframe to scipy sparse arrays

Question

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning.

Now, if my pandas' data frame consists of only numerical data, then I can simply do the following to convert the data frame to sparse csr matrix:

scipy.sparse.csr_matrix(df.values)

and if my data frame consists of ordinal data types, I can handle them using LabelEncoder

from collections import defaultdict
d = defaultdict(LabelEncoder)     
fit = df.apply(lambda x: d[x.name].fit_transform(x))

Then, I can again use the following and the problem is solved:

scipy.sparse.csr_matrix(df.values)

Categorical variables with a low number of values is also not a concern. They can easily be handled using pd.get_dummies (Pandas or Scikit-Learn versions).

My main concern is for categorical variables with a large number of values.

The main problem: How to handle categorical variables with a large number of values?

pd.get_dummies(train_set, columns=[categorical_columns_with_large_number_of_values], sparse=True)

takes a lot of time.

This question seems to be giving interesting directions, but, it is not clear whether it handles all the data types efficiently.

Let me know if you know the efficient way. Thanks.

CJR · Accepted Answer · 2020-05-19 14:50:11Z

1

You can convert any single column to a sparse COO array very easily with factorize. This will be MUCH faster than building a giant dense dataframe.

import pandas as pd
import scipy.sparse

data = pd.DataFrame({"A": ["1", "2", "A", "C", "A"]})

c, u = pd.factorize(data['A'])
n, m = data.shape[0], u.shape[0]

one_hot = scipy.sparse.coo_matrix((np.ones(n, dtype=np.int16), (np.arange(n), c)), shape=(n,m))

You'll get something that looks like this:

>>> one_hot.A
array([[1, 0, 0, 0],
       [0, 1, 0, 0],
       [0, 0, 1, 0],
       [0, 0, 0, 1],
       [0, 0, 1, 0]], dtype=int16)

>>> u
Index(['1', '2', 'A', 'C'], dtype='object')

Where rows are your dataframe rows and columns are the factors of your column (u will have labels for those columns in order)

answered May 19, 2020 at 14:50

CJR

3,9872 gold badges13 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

learner Over a year ago

How to combine multiple columns in one sparse matrix, then? ML models take in one sparse matrix.

learner Over a year ago

This solution does not work. For my single column with 14350959 unique values and 133267714 rows, it says: MemoryError: Unable to allocate 3.40 PiB for an array with shape (133267714, 14350959) and data type int16. Handling multiple columns is another problem.

CJR Over a year ago

You could vstack your encoded arrays but you should very carefully consider what the encoding means and how it works with your ML method. You do not have the memory to make a dense representation of this array and it will fail if you call array.A.

learner Over a year ago

yes, I see your point. But, I think you meant hstack (scipy.sparse.hstack), no?

Collectives™ on Stack Overflow

Converting pandas dataframe to scipy sparse arrays

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related