Spark DF: Split array to multiple rows

Question

I created spark dataframe using mongodata (in databricks using Python notebook)

I need to convert this dataframe as

How can I do this?

could you add output of .printSchema()? this sal column is just a string wirth new lines? — chlebek
– chlebek, Commented Nov 9, 2019 at 5:28
You can try pyspark.sql.functions.explode if sal column entries are arrays. — QuantStats
– QuantStats, Commented Nov 9, 2019 at 5:33
@Mahesh Gupta ``` spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri",constring).load() ``` — Raju
– Raju, Commented Nov 9, 2019 at 5:46

QuantStats · Accepted Answer · 2019-11-09 06:00:27Z

3

Here is one proposed solution. You can organize your sal field into arrays using $concatArrays in MongoDB before exporting it to Spark. Then, run something like this

#df
#+---+-----+------------------+
#| id|empno|               sal|
#+---+-----+------------------+
#|  1|  101|[1000, 2000, 1500]|
#|  2|  102|      [1000, 1500]|
#|  3|  103|      [2000, 3000]|
#+---+-----+------------------+

import pyspark.sql.functions as F

df_new = df.select('id','empno',F.explode('sal').alias('sal'))

#df_new.show()
#+---+-----+----+
#| id|empno| sal|
#+---+-----+----+
#|  1|  101|1000|
#|  1|  101|2000|
#|  1|  101|1500|
#|  2|  102|1000|
#|  2|  102|1500|
#|  3|  103|2000|
#|  3|  103|3000|
#+---+-----+----+

edited Nov 9, 2019 at 6:00

answered Nov 9, 2019 at 5:54

QuantStats

1,5061 gold badge10 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark DF: Split array to multiple rows

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related