SoFunction
Updated on 2025-04-14

Sharing of memory management mechanism and optimization techniques in Python

Remember the first time you wrote Python code? The pleasure of creating variables at will and new objects without restraint is simply addictive! However, when the code runs slowly and the memory usage soars, you will realize:

"Eh? Why does Python still eat so much memory?!"

Today, let’s talk about Python’s memory management, helping you write more efficient and optimized memory usage Python code!

1. Python's memory management mechanism

Python internal useReference CountingandGarbage Collection (GC)mechanism to manage memory.

1.1 Reference Count

Each object has a "counter" that records how many variables it is referenced. Once the reference count is zeroed, Python immediately releases the object's memory.

import sys

a = []  # Create a list objectprint((a))  # Output 2 (because () will also add an additional reference)
b = a  # Variable b also points to the same listprint((a))  # Output 3
del a
print((b))  # Output 2
del b  # Reference count is reset to zero, memory is freed

Notice：()The result is 1 more than you think, because it itself will also create a temporary reference!

1.2 Garbage Collect (GC)

Python adoptionGeneration recycling, the object is divided into three generations:The new generation, the middle generation, the old generation. Garbage recycling is mainly aimed at the situation of circular references.

import gc

class A:
    def __init__(self):
         = None

obj1 = A()
obj2 = A()
 = obj2
 = obj1  # Form a circular reference
del obj1, obj2  # The reference count is not reset to zero, Python needs GC to clean it
()  # Manually trigger garbage collection

2. Python memory optimization skills

2.1 Restrict object properties using __slots__

By default, Python objects use dynamic dictionary (__dict__) Store attributes, occupying a lot of memory. If your class attribute is fixed, you can use it__slots__Optimization, Sister Hua mentioned it in other articles before.

class NormalClass:
    pass

class SlotClass:
    __slots__ = ['name', 'age']  # Only two attributes, name and age, are allowed
obj1 = NormalClass()
 = "Sister Hua"
 = 18
 = "female"  # Allow dynamic addition of new attributes
obj2 = SlotClass()
 = "Sister Hua"
 = 18
# = "Female" # ❌ AttributeError: 'SlotClass' object has no attribute 'gender'

__slots__Will make Python no longer create objects__dict__, thereby reducing memory usage.

2.2 Avoid unnecessary temporary variables

The Python interpreter caches common objects, such asSmall integer（-5arrive256In Python 3.9 and earlier), and someShort string。

But after Python 3.10+, the cache range of integersProbably bigger, the specific behavior depends on Python implementation.

# May be cached (specific scope depends on Python version)a = 256
b = 256
print(a is b)  # True

# May not be cacheda = 257
b = 257
print(a is b)  # 3.9 Previously False, 3.10+ may be True

in conclusion：Python Small integers will be cached, but the specific scope depends on the Python version, it is not recommended to rely too much on this feature!

2.3 Use generator instead of list

If you only needGet data one by one, instead of loading all data at once, please useGeneratorInstead of list.

# How to take up a lot of memorynums = [i for i in range(10**6)]

# Better way (lazy loading)def num_generator():
    for i in range(10**6):
        yield i

gen = num_generator()

Why?The generator will not store all data into memory at once, but each timeyieldA value can greatly reduce memory usage!

2.4 Use array instead of lists to store large amounts of numerical values

If you need to store a large number of values, usearrayModule comparisonlistMore memory savings.

import array

# Create an array that stores type int, which saves more memory than listsarr = ('i', range(10**6))

2.5 Use deque to replace lists for queue operations

More efficientHead insertion and deletionOperation,listofpop(0)andinsert(0, x)Better.

from collections import deque

dq = deque(range(10**6))
(-1)  # O(1) Complexity
# And (0, -1) is O(n), and the performance gap is obvious under large-scale data

3. Release unused memory

Manually release variables

Python adoptionAutomatic garbage collection, but if you want to actively release the large object, it is recommended to usedelAnd call()。

import gc

data = [i for i in range(10**6)]
del data  # Delete variables()  # Force trigger garbage collection

In big data processing, this method canSignificantly reduce memory usage！

Summarize

Python uses reference counting + garbage collection to manage memory.

Use __slots__ to save the object's property storage space.

Avoid unnecessary temporary variables, Python caches small integers, but the scope depends on the Python version.

Replace lists with generators to save memory!

Use array instead of list to store a large number of numeric values ​​to improve memory efficiency.

Use deque instead of list for queue operation to improve performance.

Manually release the large object and clean it in time using del + ().

This is the article about sharing memory management mechanism and optimization techniques in Python. For more related Python memory management and optimization content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!