Pyspark Cast StructType as ArrayType<StructType>

Question

I have a dataframe df1 with a column col1 that has structure :

StructField(recipientResource,ArrayType(StructType(List(StructField(resourceId,StringType,true),StructField(type,StringType,true))),true),true)

and another dataframe df2 with col1 that has structure:

StructField(recipientResource,StructType(List(StructField(resourceId,StringType,true),StructField(type,StringType,true))),true)

Inorder to union df1.union(df2), I was trying to cast the column in df2 to convert it from StructType to ArrayType(StructType), however nothing which I tried has worked out.

Can anyone suggest how to go about the same. I'm new to pyspark, any help is appreciated.

array<struct<...>> and struct<...> are two completely different objects - you cannot cast one into another. You could add wrapping array if that's what you mean, like select(array(struct_column)). — Alper t. Turker
– Alper t. Turker, Commented May 10, 2018 at 18:19
An minimal reproducible example with a small sample of your dataframes and the desired output would be helpful. See more on how to create good reproducible apache spark dataframe examples. — pault
– pault, Commented May 10, 2018 at 18:35

Sourya_cool · Accepted Answer · 2020-07-26 13:28:04Z

1

Here is a simple solution using array() function:

Input:

df1 (with ArrayType(StructType()) column):

df2 (with StructType() column):

Code:

df2=(df2
     .withColumn('recipientResource',array(col('recipientResource'))) #convert StructType() column to ArrayType(StructType()) column
    )

Output:

Modified df2:

df3 (output after union of df1 and df2):

answered Jul 26, 2020 at 13:28

Sourya_cool

537 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark Cast StructType as ArrayType<StructType>

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related