Fastest way to store 3D numpy array in a loop

Question

I need to store a numpy array of shape (2000,720,1280) which is created in every loop. My code looks like:

    U_list = []   
      
    for N_f in range(N): 
             U = somefunction(N_f)
             U_list.append(U)
             del U

So I delete the matrix U in every loop because my RAM get full.

Is this a good method to store the matrix U or would you recommend another solution? I compare the code to matlab and matlab need the half time to compute. I think the storage of U in a list could be the reason.

To clarify: your U arrays are each 2000x720x1280? What's N? One option may be to preallocate a single large NumPy array and refactor somefunction to write directly to a slice of that array, rather than appending in a loop. — nneonneo
– nneonneo, Commented Apr 7, 2022 at 17:53
N is in the order of 5. I should mention that I only need the first 10 matrices of U, i.e. U[:10,:,:]. So, I store a matrix of size 10x720x1280 in every loop. But this is still very slow. — Archimedes_91
– Archimedes_91, Commented Apr 7, 2022 at 18:19
Computing 2000x720x1280 results is going to be very expensive; why do you compute so much data if you only need the first 10 elements of U? Is there maybe a cleverer way to write somefunction so it just computes the data you're interested in? — nneonneo
– nneonneo, Commented Apr 7, 2022 at 18:21
I already have done this. The storage of data is still very slow. — Archimedes_91
– Archimedes_91, Commented Apr 7, 2022 at 18:33
I think you're going to have to provide a bit more code. Just appending to a list in Python is very fast, especially if that list is only going to have ~N=5 elements in it. The elements in the list will just be references to the objects. — nneonneo
– nneonneo, Commented Apr 7, 2022 at 19:13

Will.Evo · Accepted Answer · 2022-04-07 17:58:01Z

1

Using this method will tell you if you are able to store the total U arrays right out the gate. If N is so large that you can't make the results numpy array, you'll have to get creative. Maybe save every 20 into a pickle file or something.

import numpy as np

N = 20
shape = (2000, 720, 1280)
#Make sure to match the dtype returned by somefunction
results = np.zeros((N, *shape)) 
for N_f in range(N):
    results[N_f] = somefunction(N_f)

answered Apr 7, 2022 at 17:58

Will.Evo

1,18717 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Fastest way to store 3D numpy array in a loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related