How do I convert a WrappedArray column in spark dataframe to Strings?

Question

I am trying to convert a column which contains Array[String] to String, but I consistently get this error

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 78.0 failed 4 times, most recent failure: Lost task 0.3 in stage 78.0 (TID 1691, ip-******): java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Ljava.lang.String;

Here's the piece of code

val mkString = udf((arrayCol:Array[String])=>arrayCol.mkString(","))  
val dfWithString=df.select($"arrayCol").withColumn("arrayString",
      mkString($"arrayCol"))

zero323 · Accepted Answer · 2017-01-19 23:32:12Z

31

WrappedArray is not an Array (which is plain old Java Array not a natve Scala collection). You can either change signature to:

~~import scala.collection.mutable.WrappedArray (arrayCol: WrappedArray[String]) => arrayCol.mkString(",")~~

or use one of the supertypes like Seq:

(arrayCol: Seq[String]) => arrayCol.mkString(",")

In the recent Spark versions you can use concat_ws instead:

import org.apache.spark.sql.functions.concat_ws

df.select(concat_ws(",", $"arrayCol"))

edited Jan 19, 2017 at 23:32

answered Dec 31, 2015 at 0:16

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

bdguy Over a year ago

I tried to use WrappedArray but that type wasnt recognized. Seq works fine

Burt · Accepted Answer · 2017-11-20 14:14:52Z

2

The code work for me:

df.select("wifi_ids").rdd.map(row =>row.get(0).asInstanceOf[WrappedArray[WrappedArray[String]]].toSeq.map(x=>x.toSeq.apply(0)))

In your case,I guess it is:

val mkString = udf(arrayCol=>arrayCol.asInstanceOf[WrappedArray[String]].toArray.mkString(","))  
val dfWithString=df.select($"arrayCol").withColumn("arrayString",mkString($"arrayCol"))

answered Nov 20, 2017 at 14:14

Burt

1,6001 gold badge13 silver badges10 bronze badges

Collectives™ on Stack Overflow

How do I convert a WrappedArray column in spark dataframe to Strings?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related