I would like to convert multiple array time columns in a dataframe to string. Can someone please help?
I have dataframewith different types of element.Some number/some array. I want to convert only array columns to string and the rest should be as it is.
Expected Output: Expected Output:
You can use array_join transformation in pyspark. https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.array_join.html#pyspark.sql.functions.array_join
I used something like this and that gave me the results:
selectionColumns = [F.coalesce(i[0], F.array()).alias(i[0]) if 'array' in i[1] else i[0] for i in df_grouped.dtypes ]
dfForExplode = df_grouped.select(*selectionColumns)
arrayColumns = [ i[0] for i in dfForExplode.dtypes if 'array' in i[1] ]
for col in arrayColumns:
df_grouped=df_grouped.withColumn(col,F.concat_ws(' || ',df_grouped[col]))
you can use concat_ws
>>> from pyspark.sql.functions import col, concat_ws
>>> data_df.show()
+---+---------+---------+---+---+---------------+
| _1| _2| _3| _4| _5| _6|
+---+---------+---------+---+---+---------------+
| 1|[a, b, c]|[c, d, e]| 10| 20| [a, b]|
| 2|[d, f, h]| [s, c]| 11| 21|[f, g, h, j, k]|
| 3|[a, f, g]|[r, t, y]| 12| 22| [g, h]|
+---+---------+---------+---+---+---------------+
>>> df2 = data_df.withColumn("_2",concat_ws(",",col("_2"))).withColumn("_3",concat_ws(",",col("_3"))).withColumn("_6",concat_ws(",",col("_6")))
>>> df2.show()
+---+-----+-----+---+---+---------+
| _1| _2| _3| _4| _5| _6|
+---+-----+-----+---+---+---------+
| 1|a,b,c|c,d,e| 10| 20| a,b|
| 2|d,f,h| s,c| 11| 21|f,g,h,j,k|
| 3|a,f,g|r,t,y| 12| 22| g,h|
+---+-----+-----+---+---+---------+
>>> df2.printSchema()
root
|-- _1: long (nullable = true)
|-- _2: string (nullable = false)
|-- _3: string (nullable = false)
|-- _4: long (nullable = true)
|-- _5: long (nullable = true)
|-- _6: string (nullable = false)
