Python memory usage when a variable is used for different tasks

Question

I am working in an ETL pipeline with pandas and I am exceding the memory usage of my computer.

I am reading of memory usage in Python and I don´t understand how memory usage works when I create a pandas Dataframe and I assign a name for this Dataframe and I use the same name to do some transformation or adding more columns to it.

For example:

df = pd.DataFrame(
{
'column1': [1,2]
,'column1': ['a','b']})

If now I want to add another column to this Dataframe:

df['column3'] = 1

The memory that is being used for this first df Dataframe is replaced for this new df Dataframe or now python is using memory for both Dataframes?

What happens if then I want to remove one of the columns?:

df = df.drop(columns = {'column1'})

Davide Anghileri · Accepted Answer · 2020-11-22 19:45:36Z

1

Reading the pandas documentation it says:

All pandas data structures are value-mutable (the values they contain can be altered) but not always size-mutable. The length of a Series cannot be changed, but, for example, columns can be inserted into a DataFrame.

and also:

Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects

Also, if you inspect all your in scope variables with the dir() command you can see that there is only one definition of the DataFrame.

To conclude, it seems to me that Python does not create a copy of your DataFrame and it only saves one copy when you add/remove a column. Furthermore, if you would like to create a copy of a DataFrame that actually copies all the values in another variable you should use the .copy() function.

answered Nov 22, 2020 at 19:45

Davide Anghileri

9416 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python memory usage when a variable is used for different tasks

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related