Python Multiprocessing Freeing up memory used by shared array

Question

Is there a way to deallocate the memory used by a shared Multiprocessing Array ? Here is a code snippet that creates a memory block of approx 1 GiB of shared memory which is then blocked for the remainder of the script. Is there a way to free up that memory during the same script run ?

I tried deleting all references and calling the garbage collector via gc.collect() but that seems to have no effect whatsoever on the Memory usage of the script during runtime.

import psutil
import gc
import ctypes as cty
import multiprocessing as mpc
import os

def bytes_to_GiB_MiB(n_bytes):
    return n_bytes/2**30, (n_bytes%(2**30))/2**20
    
def print_mem_usage():
    process = psutil.Process(os.getpid()) 
    print "Script Mem Usage {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(process.memory_info().rss))
    return process.memory_info().rss    
    
if __name__=="__main__":
    t_N = 5000
    N=150
    Shape_spl = (t_N+4, N,N) 
    mem0 = print_mem_usage()
    C_shared = mpc.Array(cty.c_double, N*N*(t_N+4))
    mem1 = print_mem_usage()
    print "Mem Difference is {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(mem1-mem0))
    del C_shared
    gc.collect()
    # Do something here such that the memory used by C_shared is free again.
    mem2 = print_mem_usage()
    print "Mem Difference is {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(mem2-mem0))

The problem is resolved in Python 3.8.1 (default, Jan 8 2020, 22:29:32) \n[GCC 7.3.0]. I tested the script initially with Python 2.7, there the memory is not released. With the new python the memory is released. I am still interested if there is a workaround for Python 2.7. — Hans Wurst
– Hans Wurst, Commented Jul 2, 2020 at 14:53

raphael · Accepted Answer · 2021-03-05 16:23:06Z

1

... for the record... This seems to be related to this bug in python <= 3.7
(as already mentioned, it is supposed to be fixed in python 3.8):
https://bugs.python.org/issue32759

There's also a crude "fix" suggested in the discussion which seems to work for me with python 3.7:
(not sure if this also works in python 2.7)

"... delete the globe _heap and recreate a new one"
mp.heap.BufferWrapper._heap = mp.heap.Heap()

answered Mar 5, 2021 at 16:23

raphael

3,1891 gold badge20 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Multiprocessing Freeing up memory used by shared array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related