Python Pandas dataframe find missing values

Question

I'm trying to find missing values and then drop off missing values. Tried looking for the data online but can't seem to find the answer.

Extracted Dataframe:

In the df, for 1981 and 1982, it should be '-', i.e. missing values. I would like to find the missing values then drop off the missing values.

Exported Dataframe using isnull:

I used df.isnull() but in 1981 and 1982, it's detected as 'False' which means there's data. But it should be '-', therefore considered as missing values.

I had pasted my code below. What am I missing out?

import pandas as pd

mydf = pd.read_excel('abc.xlsx', sep='\t')

df1 = mydf.set_index('Variables')
df = df1[0:10]
print(df)
print(df.isnull())

Gonçalo Peres · Accepted Answer · 2020-07-22 19:54:56Z

7

The question has two points: finding which columns have missing values and drop those values.

To find the missing values on a dataframe df

missing = df.isnull().sum()
print(missing)

To drop those missing values, apart from @jezrael's consideration, if that doesn't help, I suggest you to use dropna:

Drop the rows where all elements are missing.

df.dropna(how='all')

Drop the columns where at least one element is missing.

df.dropna(axis='columns')

answered Jul 22, 2020 at 19:54

Gonçalo Peres

13.9k5 gold badges73 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

Missing values are not -.

So for missing values use na_values parameter in read_excel for converting - to missing values NaNs:

mydf = pd.read_excel('abc.xlsx', sep='\t', na_values='-')

na_values : scalar, str, list-like, or dict, default None

Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN: '', '#N/A', '#N/A N/A', '#NA', '-1.#IND', '-1.#QNAN', '-NaN', '-nan', '1.#IND', '1.#QNAN', 'N/A', 'NA', 'NULL', 'NaN', 'n/a', 'nan', 'null'.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jan 11, 2020 at 13:49

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Comments

Dinesh vishe · Accepted Answer · 2024-04-14 15:11:01Z

0

features_with_na=[features for features in df.columns if df[features].isnull().sum()>=1] for feature in features_with_na: print(feature,np.round(df[feature].isnull().mean()*100,5), '% missing values')

answered Apr 14, 2024 at 15:11

Dinesh vishe

3,6581 gold badge27 silver badges26 bronze badges

Collectives™ on Stack Overflow

Python Pandas dataframe find missing values

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related