Convert Dask Dataframe to Spark dataframe using Python

Question

I want to convert Dask Dataframe to Spark Dataframe.

Let's consider this example:

import dask.dataframe as dd
dask_df = dd.read_csv("file_name.csv")

# convert dask df to spark df
spark_df = spark_session.createDataFrame(dask_df)

But this is not working. Is there any alternative to do this. Thanks in advance.

Powers · Accepted Answer · 2021-10-08 12:51:59Z

3

Writing the Spark DataFrame to disk with Dask and reading it with Spark is the best for bigger datasets.

Here's how you can convert smaller datasets.

pandas_df = dask_df.compute()
pyspark_df = spark.createDataFrame(pandas_df)

I don't know of an in-memory way to convert a Dask DataFrame to a Spark DataFrame without a massive shuffle, but that'd certainly be a cool feature.

answered Oct 8, 2021 at 12:51

Powers

19.5k12 gold badges113 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mdurant · Accepted Answer · 2021-02-25 20:19:42Z

1

You best alternative is to save the dataframe to files, e.g., the parquet format: dask_df.to_parquet(...). If your data is small enough, you could load it entirely into the client and feed the resultant pandas dataframe to Spark.

Although it's possible to co-locate spark and dask workers on nodes, they will no be in direct communication with each other, and streaming large data via the client is not a good idea.

answered Feb 25, 2021 at 20:19

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Collectives™ on Stack Overflow

Convert Dask Dataframe to Spark dataframe using Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related