replace null values in string type column with zero PySpark

Question

I need to replace null values in string type columns to be 0. Data looks like this:

df.groupBy('content').count().show() 
+---------------+------+
|        content| count|
+---------------+------+
|         videos|   754|
|      food-news| 76151|
|           null|    39|
|             uk| 23879|

I have tried this:

df.na.fill(0).show()

But this piece of code only takes care of int type columns. How can I replace it for string type columns?

Thank you.

mck · Accepted Answer · 2020-12-08 16:13:26Z

1

Fill with a string '0' too:

df = df.na.fill(0).na.fill('0')

edited Dec 8, 2020 at 16:13

answered Dec 8, 2020 at 16:07

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Chique_Code Over a year ago

After applying your peice of code, I ran df.groupBy('my_column').count().show() I still see null in the list...

mck Over a year ago

@Chique_Code Did you do df = df.na.fill('0')? Simply doing df.na.fill('0') wil not modify df. I've updated my answer to reflect this.

Chique_Code Over a year ago

Ah! That did the trick. Sorry about that. I have accepted and upvoted the answer.

user19195895 · Accepted Answer · 2022-06-24 06:12:48Z

0

In below code all int values will be replaced by 0 and string values to ' '(blank).

df = df.na.fill(0).na.fill(' ')

answered Jun 24, 2022 at 6:12

user19195895

414 bronze badges

Collectives™ on Stack Overflow

replace null values in string type column with zero PySpark

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related