3

I have 166600 numpy files, I want to put them into one numpy file: file by file, I mean that the creation of my new big file must from the begin: the first file must be read and written in the file, so the big file contains only the first file, after that I need to read and write the second file, so the big file contains the first two files.

import matplotlib.pyplot as plt 
import numpy as np
import glob
import os, sys
fpath ="path_Of_my_final_Big_File"
npyfilespath ="path_of_my_numpy_files"   
os.chdir(npyfilespath)
npfiles= glob.glob("*.npy")
npfiles.sort()
all_arrays = np.zeros((166601,8000))
for i,npfile in enumerate(npfiles):
    all_arrays[i]=np.load(os.path.join(npyfilespath, npfile))
np.save(fpath, all_arrays)
Improve this question
1

1 Answer 1

Reset to default
5

If I understand your questions correctly, you can use numpy.concatenate for this:

import matplotlib.pyplot as plt 
import numpy as np
import glob
import os, sys
fpath ="path_Of_my_final_Big_File"
npyfilespath ="path_of_my_numpy_files"   
os.chdir(npyfilespath)
npfiles= glob.glob("*.npy")
npfiles.sort()
all_arrays = []
for i, npfile in enumerate(npfiles):
    all_arrays.append(np.load(os.path.join(npyfilespath, npfile)))
np.save(fpath, np.concatenate(all_arrays))

Depending on the shape of your arrays and the intended concatenation, you might need to specify the axis parameter of concatenate.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

But numpy.savez will give me a zipped file, I need just a numpy uncompressed file
The doc states "Save several arrays into a single file in uncompressed .npz format.". It just uses the zipped file so that all individual files can individually be found in the archive. Or do you want to save the concatenation of all your arrays as a single array?
I want to save the concatenation of all your arrays as a single array in the same numpy file. I mean that my file will include just one big array which contains many arrays.
That wasn't clear from your question. So please update it in that direction.
I gives me this error: np.save(fpath, np.append(all_arrays)) TypeError: append() takes at least 2 arguments (1 given)
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.