Remove Array Column from Array pyspark

Question

Consider I have the following data structure in a pyspark dataframe:

arr1:array
   element:struct
     string1:string
     arr2:array
         element:string
     string2: string

How can I remove the arr2 from my dataframe?

use to_json + from_json, see one similar post: stackoverflow.com/questions/58243292 — jxc
– jxc, Commented Oct 15, 2019 at 17:58

pissall · Accepted Answer · 2019-10-16 03:30:24Z

0

You can use the drop function only. The way to select the nested columns is with .

Like window.start and window.end. You can access your arr2 as arr1.element.arr2.

df.drop(df.element.arr2)

answered Oct 16, 2019 at 3:30

pissall

7,4442 gold badges29 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1