In short there are three memory management mechanisms in python
1) Reference Counting
2) Garbage collection
3) Memory Pool
Next, we'll explain these three management mechanisms in more detail
1, quote count:
Reference counting is a very efficient memory management tool. When a pyhton object is referenced its reference count is increased by 1, when it is no longer referenced its reference count is decreased by 1, and when the reference count equals 0, the object is deleted.
2, garbage collection (this is a very important knowledge):
① Citation Count
Reference counting is also a garbage collection mechanism and is one of the most intuitive and simple garbage collection techniques.
The core of every object in Python is a structure PyObject, which has an internal reference count ob_refcnt, when the reference count of an object in Python is 0. This means that there are no references pointing to the object, and the object becomes garbage to be recycled.
For example: when an object is created, its reference count will be +1, when the object is referenced, the count will continue to increase, and when the object referencing it is deleted, its reference count will decrease. Until it becomes 0, at which point the garbage collection mechanism will recycle it. But once a circular reference occurs, we have to take a new approach.
② Marker removal
Marker removal is used to solve the problem of circular references, which are only generated in container objects, such as dictionaries, meta-ancestors, lists, etc. First of all, in order to keep track of the object, each container object needs to maintain two additional pointers, which are used to form a chained list of container objects, the pointers point to the front and back of the two container objects, so that you can remove the circular references to the object, and then you can derive the effective count of the two objects.
Code Real Chestnut
QA: Why do you need these two linked tables?
The reason for dissecting into two linked lists is based on the consideration that there may be objects in the current unreachable that are directly or indirectly referenced by the objects in the root linked list, which cannot be recycled, and that once such an object is found during the marking process, it will be moved from the unreachable linked list to the root linked list; when the marking is completed, all the objects left in the unreachable linked list are literally garbage objects. When the marking is done, all the objects left in the unreachable chain are garbage objects in name only, and the next garbage collection can be restricted to the unreachable chain.
③ Split-generation recycling
Understanding the classification of recycling, first of all, to understand, GC threshold, the so-called threshold is a critical point of value.
As your program runs, the Python interpreter keeps track of newly created objects, as well as objects that are released because their reference counts are zero. Theoretically, this is how creation == release count should look like.
Generation recycling idea divides objects into three generations (generation 0,1,2)
0 for juvenile subjects.
1 on behalf of the young subjects.
2 Represents elderly subjects.
According to the weak generation hypothesis (the younger the object the more likely it is to die out, older objects usually survive longer.)
The newborn object is put into generation 0. If the object survives a gc garbage collection in generation 0, it is put inside generation 1 (and it is upgraded). If the object inside generation 1 survived a gc garbage collection in generation 1, it was put inside generation 2.
Since the last generation 0 gc, if the number of allocated objects minus the number of released objects is greater than threshold0, then a gc garbage collection check is performed on the objects in generation 0.
Since the last generation 1 gc, if the number of times generation 0 has been gc garbage collected is greater than threshold1, then a gc garbage collection check is performed on the objects in generation 1.
Since the last generation 2 gc, if the number of times generation 1 has been gc garbage collected is greater than threshold2, then a gc garbage collection check is performed on the objects in generation 2.
The threshold triggered by each generation of garbage collection by gc can be set by yourself.
3, Memory Pool
- Python's memory mechanism takes the shape of a pyramid, with layers -1 and -2 operated by the operating system.
- Layer 0 is C's malloc, free, and other memory allocation and release functions that perform operations
- Layers 1 and 2 are memory pools, implemented by the python interface function, PyMem_Malloc, which allocates memory directly from this layer for objects smaller than 256k.
- Layer 3 is the top layer, which is our direct manipulation of python objects
Python performs a lot of malloc and free operations during runtime, and frequently switches between the user state and the kernel state, which seriously affects Python's execution efficiency. In order to speed up Python's execution efficiency, Python introduces a memory pooling mechanism to manage the requesting and releasing of small chunks of memory.
4, means of tuning
1. Manual garbage collection
2. Avoid circular references (manually un-cycling references and using weak references)
3. Raise garbage collection thresholds
The above is a small introduction to the python memory management and garbage collection mechanism detailed integration, I hope to help you, if you have any questions please leave me a message, I will reply to you in a timely manner. Here also thank you very much for your support of my website!