Convert pandas DataFrame to record array without the extra column

Question

I have the following code

import numpy as np
import pandas as pd

test_array = np.array([(1, 2, 3), (4, 5, 6)], 
                      dtype={'names': ('a', 'b', 'c'), 'formats': ('f8', 'f8', 'f8')})
test_df = pd.DataFrame.from_records(test_array)
test_df.to_records().view(np.float64).reshape(test_array.shape + (-1, ))

I expect a view on the original test_array to be returned, with shape (2, 3), however, I get this (2, 4) array.

rec.array([[0.e+000, 1.e+000, 2.e+000, 3.e+000],
           [5.e-324, 4.e+000, 5.e+000, 6.e+000]],
          dtype=float64)

Where did the extra column, column 0, come from?

Edit: I've just learned I can use DataFrame.values() to do the same thing, but I remain curious why this behavior exists.

cs95 · Accepted Answer · 2019-05-23 17:35:54Z

4

If you need a record array, use np.rec.fromrecords:

np.rec.fromrecords(test_df, names=[*test_df])
# rec.array([(1., 2., 3.), (4., 5., 6.)],
#          dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])

My tests show that this is faster than df.to_records by some.

answered May 23, 2019 at 17:35

cs95

406k106 gold badges744 silver badges798 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

yatu · Accepted Answer · 2019-05-23 17:27:42Z

2

to_records is capturing the index too. Note that this is stated in the docs:

Index will be included as the first field of the record array if requested

If you want to exlude it simply set index=False.

Although in your case you can simply use to_numpy (or values):

test_df.to_numpy().view(np.float64).reshape(test_array.shape + (-1, ))

array([[1., 2., 3.],
       [4., 5., 6.]])

answered May 23, 2019 at 17:27

yatu

88.7k12 gold badges93 silver badges148 bronze badges

3 Comments

mnosefish Over a year ago

This is a bit of a contrived example, but I have some code that expects a record array because it uses the dtype names internally. Is there a way to do this and maintain those values.

mnosefish Over a year ago

Oh! I missed this before. I had created a separate function to remove the index and apparently it didn't work very well.

yatu Over a year ago

So perhaps you could go with to_records setting index=False? @mnosefish

kudeh · Accepted Answer · 2019-05-23 17:28:53Z

2

Set the index=False in to_records:

test_df.to_records(index=False).view(np.float64).reshape(test_array.shape + (-1, ))

answered May 23, 2019 at 17:28

kudeh

9131 gold badge6 silver badges17 bronze badges

Collectives™ on Stack Overflow

Convert pandas DataFrame to record array without the extra column

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related