You can pad your array. Padding will extend your array with the desired boundary conditions (see the mode parameter for all possible options):
>>> A = np.array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
>>> N = 5 # 5x5 windows
>>> B = np.pad(A, N//2, mode='reflect')
>>> B
array([[10, 9, 8, 9, 10, 11, 10, 9],
[ 6, 5, 4, 5, 6, 7, 6, 5],
[ 2, 1, 0, 1, 2, 3, 2, 1],
[ 6, 5, 4, 5, 6, 7, 6, 5],
[10, 9, 8, 9, 10, 11, 10, 9],
[14, 13, 12, 13, 14, 15, 14, 13],
[10, 9, 8, 9, 10, 11, 10, 9],
[ 6, 5, 4, 5, 6, 7, 6, 5]])
As you can see, the original array is in the center of the matrix, padded by 2 rows and 2 columns (N//2 = 5//2 = 2, both from lef/right and bottom/top). The padded elements are reflected.
For this new array, you can access to the desired neighbours window by just normally indexing the array:
>>> x = 1; y = 1 # corresponds to number 5 in the original array
>>> B[y:y+N, x:x+N]
array([[ 5, 4, 5, 6, 7],
[ 1, 0, 1, 2, 3],
[ 5, 4, 5, 6, 7],
[ 9, 8, 9, 10, 11],
[13, 12, 13, 14, 15]])
You can choose other padding methods, mean is one of the options.
scipy.ndimage.generic_filter.