0

I have a memmapped numpy array:

arr = np.load("a.npy", mmap_mode='r')

It is bigger than my memory. For my further computation I need it in fortran order instead of C. So I can use np.asfortranarray to convert it and then use np.save to store it in a new file.

However when I do this my memory usage increases proportionally to the input and as such makes me think that an object is being created in memory, I would like it to be fully a, file to file interaction.

How can I convert a.npy into fortran order without having the full object in memory?


For example, I have array:

arr =array([[1, 2],
            [3, 4]])

This is stored on disk as follows by numpy:

fortran_order: False
01 02 03 04

I can do the following transformations:

1

np.asfortranarray(arr)
array([[1, 2],
       [3, 4]])

saved as:

fortran_order: True
01 03 02 04

2

np.asfortranarray(arr.T)
array([[1, 3],
       [2, 4]])

saved as:

fortran_order: True
01 02 03 04

3

arr.T
array([[1, 3],
       [2, 4]])

saved as:

fortran_order: True
01 02 03 04

arr.T only converts the high level accessiblity, I need the on disk ordering to be swapped (this will help with my overall task by keeping the array indexing with C in cache). This can be done by calling np.asfortranarray without arr.T however this incurs a full data copy instead of a transposed view being created and written in the transposed order.

2
  • If you want a contiguous Fortran array, then this is not possible since the array is stored contiguously using the row-major ordering while Fortran is column major. If this is fine to get a non-contiguous Fortran array (memory view of a C array with strides), then this is possible but the access will be inefficient for native Fortran accesses, and fast for native C one. The conversion to a contiguous Fortran array require a (mandatory) copy: data needs to be reordered in memory. Commented Jan 10, 2023 at 15:26
  • this related post might help you to understand how things works (though the context is different). Commented Jan 10, 2023 at 15:30

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.