Pyspark - Cast a column in a nested array

Question

I have a dataframe with the following schema:

root
 |-- Id: long (nullable = true)
 |-- LastUpdate: string (nullable = true)
 |-- Info: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- Purchase: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- Amount: long (nullable = true)
 |    |    |    |    |-- Name: string (nullable = true)
 |    |    |    |    |-- Type: string (nullable = true)

How can I select the Amount column such that I can cast it?

Tried:

df = df.withColumn("Info.Purchase.Amount", df["Info.Purchase.Amount"].cast(DoubleType()))

But got:

org.apache.spark.sql.AnalysisException: cannot resolve '`Info`.`Purchase`['Amount']'

pardeep garg · Accepted Answer · 2020-09-04 16:27:57Z

1

You can use below method to extract nested array:

df.select(col("info").getField("Purchase").getField("Amount")).show()

This will give you list of all amount column. you can cast that.

answered Sep 4, 2020 at 16:27

pardeep garg

2191 silver badge11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark - Cast a column in a nested array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related