From pandas dataframe to multidimensional numpy array for compatibility with tensorflow

Question

I have a pandas dataframe where one of the columns contains lists:

import pandas as pd
import numpy as np
(
   pd.DataFrame({
      "x": [[1, 2], [3, 4], [5, 6]]
   )}
   .assign(x = lambda data: data.x.apply(np.array))  # convert lists into numpy arrays
   .to_numpy()
   .shape  # returns (3, 1) when I was hoping for a (3,1,2)
)

I would like to pass this data into tensorflow as a 3D array, but first I need to be able to get the right shape out of it.

Many thanks!

@JvdV, I'm not a 100% sure, but I thought it should be (3,1,2): 3 rows, 1 column with 2 dimensions — IVR
– IVR, Commented Apr 3, 2020 at 7:11

JvdV · Accepted Answer · 2020-04-03 08:10:18Z

1

You could retrieve it like:

import pandas as pd
import numpy as np
npArr = np.array(pd.DataFrame({"x": [[1, 2], [3, 4], [5, 6]],
                               "y": [[1, 2], [3, 4], [5, 6]]}).values.tolist())
print(npArr.shape)

edited Apr 3, 2020 at 8:10

answered Apr 3, 2020 at 7:12

JvdV

76.8k8 gold badges48 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

IVR Over a year ago

Thank you! I have a few follow up questions: (1) I thought that the first dimension should correspond to rows, any ideas on why the first dimension is 1 (column)? E.g. if I do pd.DataFrame({"x": [1, 2, 3]}).to_numpy().shape it returns (3,1) as expected

IVR Over a year ago

And one more question: while the proposed solution makes sense, I feel like it's not quite complete for my use-case (even though it actually answers the original question): you see, I have 1000s of multidimensional variables like x and I would like to keep them all in one dataset, so would you be able to expand you answer to cover a case where the output table covers multiple columns, so instead of (1,3,2), say (2,3,2)?

IVR Over a year ago

Excellent! Thank you so much, that's exactly what I was looking for!

JvdV Over a year ago

it's based of this answer by @jezrael. I credited, so please also give credit where credit is due =)

JvdV Over a year ago

@IVR, you are welcome. Please see above comment which direct to another post that has been of great help. Also, let's clean up the comments below question/answer.

Collectives™ on Stack Overflow

From pandas dataframe to multidimensional numpy array for compatibility with tensorflow

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related