Replace NaN Values with Average of Columns - NumPy

In real-world datasets, missing values (NaN) are common and can cause errors in calculations or visualizations. One practical solution is to replace NaN values with the average of their respective columns. Below are different methods, arranged from most efficient to least efficient.

Using np.nanmean and np.take 

This vectorized approach uses NumPy functions to calculate column means while ignoring NaNs and replaces them efficiently in the original array. It is fast and memory-efficient.

Example: Here we compute the column averages and replace all NaN values with the corresponding column mean using np.take.

import numpy as np

arr = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])
col_mean = np.nanmean(arr, axis=0)
inds = np.where(np.isnan(arr))

arr[inds] = np.take(col_mean, inds[1])
print(arr)

[[1.3 2.5 3.6 6. ]
 [2.6 3.3 4.5 5.5]
 [2.1 3.2 5.4 6.5]]

Explanation:

• np.nanmean(arr, axis=0) computes the mean of each column ignoring NaNs.
• np.where(np.isnan(arr)) returns the indices of all NaN elements.
• np.take(col_mean, inds[1]) selects the column mean corresponding to each NaN.
• arr[inds] = ... replaces NaNs with the computed column averages.

Using np.ma and np.where 

This method creates a masked array where NaNs are ignored and uses np.where to build a new array with column averages replacing NaNs.

Example: Replace NaNs by computing column averages on a masked array and filling in missing values using np.where.

import numpy as np
arr = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])
res = np.where(np.isnan(arr), np.ma.array(arr, mask=np.isnan(arr)).mean(axis=0), arr)
print(res)

[[1.3 2.5 3.6 6. ]
 [2.6 3.3 4.5 5.5]
 [2.1 3.2 5.4 6.5]]

Explanation:

• np.ma.array(arr, mask=np.isnan(arr)) creates a masked array ignoring NaNs.
• .mean(axis=0) computes the column averages of unmasked elements.
• np.where(np.isnan(arr), column_means, arr) replaces NaNs with column means.

Using Python loops and zip 

This method iterates over NaN positions and replaces them with column averages. Slower than vectorized NumPy approaches but works without NumPy functions.

Example: We find NaN positions using np.where, then replace each NaN with the column mean using a loop.

import numpy as np
arr = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])
inds = np.where(np.isnan(arr))
for r, c in zip(*inds):
    arr[r, c] = np.mean(arr[~np.isnan(arr[:, c]), c])

print(arr)

[[1.3 2.5 3.6 6. ]
 [2.6 3.3 4.5 5.5]
 [2.1 3.2 5.4 6.5]]

Explanation:

• np.where(np.isnan(arr)) finds row and column indices of NaNs.
• For each NaN, np.mean(arr[~np.isnan(arr[:, c]), c]) computes the mean of the column excluding NaNs.
• Each NaN is replaced with its column average via the loop.

Using list comprehension and built-in functions

For Python lists (not NumPy arrays), you can compute column averages and replace None values using list comprehension. Slower due to Python loops.

Example: Compute column averages and replace None values with the corresponding mean for each row.

arr = [[1.3, 2.5, 3.6, None], [2.6, 3.3, None, 5.5], [2.1, 3.2, 5.4, 6.5]]
col_means = [sum(filter(None, col))/len(list(filter(None, col))) for col in zip(*arr)]
arr = [[col_means[j] if x is None else x for j, x in enumerate(row)] for row in arr]
print(arr)

[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]

Explanation:

• zip(*arr) transposes the array for column-wise processing.
• filter(None, col) removes None values to compute column mean.
• Replace None in each row with corresponding column mean using list comprehension.