Reshape a dask array (obtained from a dask dataframe column)

Question

I am new to dask and am trying to figure out how to reshape a dask array that I've obtained from a single column of a dask dataframe and am running into errors. Wondering if anyone might know of the fix (without having to force a compute)? Thanks!

Example:

import pandas as pd
import numpy as np
from dask import dataframe as dd, array as da
df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
ddf = dd.from_pandas(df, npartitions=2)

# This does not work - error ValueError: cannot convert float NaN to integer
ddf['x'].values.reshape([-1,1])

# this works, but requires a compute
ddf['x'].values.compute().reshape([-1,1])

# this works, if the dask array is created directly from a np array
ar = np.array([1, 2, 3])
dar = da.from_array(ar, chunks=2)
dar.reshape([-1,1])

Rowan_Gaffney · Accepted Answer · 2019-07-29 23:12:50Z

2

Also:

ddf['x'].to_dask_array(lengths=True).reshape([-1,1])

answered Jul 29, 2019 at 23:12

Rowan_Gaffney

4625 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mdurant · Accepted Answer · 2018-09-08 00:55:51Z

0

Unfortunately, then length of a dataframe and its pieces is generally lazy in Dask, and only computed on explicit request. That means that the array doesn't know its length or partitioning either, and so you can't reshape. The following clunky code gets around this, but I feel there should be a simpler way.

Find the chunks:

chunks = tuple(ddf['x'].map_partitions(len).compute())
size = sum(chunks)

Create a new array object with the now-known chunks and size:

a = ddf['x'].values
arr = da.Array(a.dask, a.name, chunks, a.dtype, (size,))

answered Sep 8, 2018 at 0:55

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Collectives™ on Stack Overflow

Reshape a dask array (obtained from a dask dataframe column)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related