Explode array of structs to columns in pyspark

Question

I'd like to explode an array of structs to columns (as defined by the struct fields). E.g.

    root
 |-- news_style_super: array (nullable = true)
 |    |-- element: array (containsNull = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- name: string (nullable = true)
 |    |    |    |-- sbox_ctr: double (nullable = true)
 |    |    |    |-- wise_ctr: double (nullable = true)

Should be transformed to

|-- name: string (nullable = true)
|-- sbox_ctr: double (nullable = true)
|-- wise_ctr: double (nullable = true)

How can I do this?

You are a new contributor and welcome, but this not how you ask a question. You should work on your code and put what have you done so far, here. Read the Help Center topics to learn more about what questions you can ask and what type of questions you have to avoid. Help Center, What types of questions should I avoid asking?, What topics can I ask about here?. Community will definitely be glad to help you for right questions. — Armin Taghavizad
– Armin Taghavizad, Commented Oct 12, 2019 at 8:25

mimic0 · Accepted Answer · 2019-10-15 06:50:07Z

def get_final_dataframe(pathname, df):
cur_names = pathname.split(".")
if len(cur_names) > 1:
    root_name = cur_names[0]
    delimiter = "."
    new_path_name = delimiter.join(cur_names[1:len(cur_names)])

    for field in df.schema.fields:
        if field.name == root_name:
            if type(field.dataType) == ArrayType:
                return get_final_dataframe(pathname, df.select(explode(root_name).alias(root_name)))
            elif type(field.dataType) == StructType:
                if hasColumn(df, delimiter.join(cur_names[0:2])):
                    return get_final_dataframe(new_path_name, df.select(delimiter.join(cur_names[0:2])))
                else:
                    return -1, -1
            else:
                return -1, -1

else:
    root_name = cur_names[0]
    for field in df.schema.fields:
        if field.name == root_name:
            if type(field.dataType) == StringType:
                return df, "string"
            elif type(field.dataType) == LongType:
                return df, "numeric"
            elif type(field.dataType) == DoubleType:
                return df, "numeric"
            else:
                return df, -1

return -1, -1

then,you can

key = "a.b.c.name"
# key = "context.content_feature.tag.name"
df2, field_type = get_final_dataframe(key, df1)

Collectives™ on Stack Overflow

Explode array of structs to columns in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related