Appending numpy arrays into two binary files

Question

I want to create two binary files to append numpy arrays into each one of them during a loop. I wrote the following method (I use Python 2.7):

for _ in range(5):
    C = np.random.rand(1, 5)
    r = np.random.rand(1, 5)
    with open("C.bin", "ab") as file1, open("r.bin", "ab") as file2:
        # Append to binary files
        np.array(C).tofile(file1)
        np.array(r).tofile(file2)

# Now printing to check if appending is successful
C = np.load("C.bin")
r = np.load("r.bin")
print (C)
print (r)

However, I keep getting this error:

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    C = np.load("C.bin")
  File "/anaconda/lib/python2.7/site-packages/numpy/lib/npyio.py", line 429, in load
    "Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file 'C.bin' as a pickle

I tried to fix it but I cannot see anything more. Any help is appreciated.

NOTE: I intentionally want to use np.load because later on I will be loading the dataset from the disk into a numpy array for further processing.

chgsilva · Accepted Answer · 2017-11-21 01:40:15Z

1

You should use the save method that is built in the numpy to store the array in the files. Here what your code should look like:

for _ in range(5):
    C = np.random.rand(1, 5)
    r = np.random.rand(1, 5)

    np.save('C', C)
    np.save('r', r)

    # Now printing to check if appending is successful
    C = np.load("C.npy")
    r = np.load("r.npy")
    print (C)
    print (r)
    del C, r

Please refer to the documentation https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.load.html

answered Nov 21, 2017 at 1:40

chgsilva

7687 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Medo Over a year ago

Thank you for your answer. But this is not appending, rather rewriting! I printed it outside the loop and it looks like it is only saving the last block

chgsilva Over a year ago

Hmmm I got your point @Medo , you can create an array of arrays and store it as one. Or you can use np.savez() function that allow you to store more than one item in the same file, please check the sintax at docs.scipy.org/doc/numpy-1.13.0/reference/generated/…

Medo Over a year ago

Good hint. Thanks alot

Collectives™ on Stack Overflow

Appending numpy arrays into two binary files

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related