efficient array concatenation

Question

Im trying to concatenate several hundred arrays size totaling almost 25GB of data. I am testing on 56 GB machine, but i receive memory error. I reckon the way I do my precess is ineffecient and is sucking lots of memory. This is my code:

    for dirname, dirnames, filenames in os.walk('/home/extra/AllData'): 
        filenames.sort()
    BigArray=numpy.zeros((1,200))
    for file in filenames:
        newArray[load(filenames[filea])
        BigArray=numpy.concatenate((BigArrat,newArray))

any idea, thoughts or solutions?

Thanks

Why do you need a 25Gb array in memory? It will never fit there, you'll need to process it piece by piece. — sashkello
– sashkello, Commented Oct 31, 2013 at 5:37
@sashkello I do need it in memory as my application runs in realtime. — Adham Ghazali
– Adham Ghazali, Commented Oct 31, 2013 at 5:42
The fact that it runs in real time doesn't mean you need all 25Gb all the time. I can be wrong, and maybe you have a very particular case on your hands, but I have a strong feeling that this is a poor program design choice and you need to redo it, rather than trying to fit this huge amount of data. I suggest adding details about your particular situation in your question, otherwise there is really no good answer to it besides "don't do it". — sashkello
– sashkello, Commented Oct 31, 2013 at 5:46
I think alloc a large array and fill in it is much faster than concatenate, because concatenate will copy the whole array every time. — HYRY
– HYRY, Commented Oct 31, 2013 at 6:00
As @HYRY suggests, preallocate the array first, although you will need to know the total size first. Problem is that when you do a = np.concatenate((a, b)), numpy creates an intermediate array with the concatenation of a and b, then points variable a to it, and if there are no other references to the old a it then gets garbage collected. But it requires, even if only for a split second, that you have twice as much memory as the final array. — Jaime
– Jaime, Commented Oct 31, 2013 at 7:26

Community · Accepted Answer · 2020-06-20 09:12:55Z

6

Your process is horribly inefficient. When handling such huge amounts of data, you really need to know your tools.

For your problem, np.concatenate is forbidden - it needs at least twice the memory of the inputs. Plus it will copy every bit of data, so it's slow, too.

Use numpy.memmap to load the arrays. That will use only a few bytes of memory while still being pretty efficient.

Join them using np.vstack. Call this only once (i.e. don't bigArray=vstack(bigArray,newArray)!!!). Load all the arrays in a list allArrays and then call bigArray = vstack(allArrays)
If that is really too slow, you need to know the size of the array in advance, create an array of this size once and then load the data into the existing array (instead of creating a new one every time).

Depending on how often the files on disk change, it might be much more efficient to concatenate them with the OS tools to create one huge file and then load that (or use numpy.memmap)

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Oct 31, 2013 at 11:39

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Adham Ghazali Over a year ago

I ended up pre allocating the size of the BigArray and inserting the newArrays into it.

Collectives™ on Stack Overflow

efficient array concatenation

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related