Spark DataFrame making column null value to empty

Question

I have joined two data frames with left outer join. Resulting data frame has null values. How do to make them as empty instead of null.

| id|quantity|
+---+--------
|  1|    null|
|  2|    null|
|  3|    0.04

And here is the schema

root
|-- id: integer (nullable = false)
|-- quantity: double (nullable = true)

expected output

| id|quantity|
+---+--------
|  1|        |
|  2|        |
|  3|    0.04

Alberto Bonsanto · Accepted Answer · 2016-05-17 11:06:36Z

4

You cannot make them "empty", since they are double values and empty string "" is a String. The best you can do is leave them as nulls or set them to 0 using fill function:

val df2 = df.na.fill(0.,Seq("quantity"))

Otherwise, if you really want to have empty quantities, you should consider changing quantity column type to String.

edited May 17, 2016 at 11:06

Alberto Bonsanto

18.1k10 gold badges67 silver badges93 bronze badges

answered May 17, 2016 at 7:22

TheMP

8,4479 gold badges46 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark DataFrame making column null value to empty

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related