Explaining the use of magic methods in Python in detail

Magic methods in python are special methods that allow you to add "magic" to a class, often named with two underscores.

Python's magic methods, also known as dunder (double underscore) methods. Most of the time, we use them for simple things like constructors (__init__), string representations (__str__, __repr__) or arithmetic operators (__add__/__mul__). In fact, there are many more that you may not have heard of but that work well, and in this article, we'll organize those magic methods!

Iterator size

We all know the __len__ method, which can be used to implement the len() function on a container class. But what if you want to get the length of the object of the class that implements the iterator?

it = iter(range(100))
 print(it.__length_hint__())
 # 100
 next(it)
 print(it.__length_hint__())
 # 99
 
 a = [1, 2, 3, 4, 5]
 it = iter(a)
 print(it.__length_hint__())
 # 5
 next(it)
 print(it.__length_hint__())
 # 4
 (6)
 print(it.__length_hint__())
 # 5

All you need to do is implement the __length_hint__ method, which is a built-in method (not a generator) on the iterator, as you can see above, and also supports dynamic length changes. But, as his name suggests, it's just a hint and is not guaranteed to be completely accurate: for list iterators, it gives accurate results, but for other iterators it's not sure. But even if it's not accurate, it can help us get the information we need, as explained in PEP 424

length_hint must return an integer (else a TypeError is raised) or NotImplemented, and is not required to be accurate. It may return a value that is either larger or smaller than the actual size of the container. A return value of NotImplemented indicates that there is no finite length estimate. It may not return a negative value (else a ValueError is raised).

metaprogramming

Most of the seldom-seen magical methods have to do with metaprogramming, and while metaprogramming may not be something we need to use every day, there are some handy tricks to use it.

One such trick is to use __init_subclass__ as a shortcut to extend the functionality of the base class without having to deal with metaclasses: .

class Pet:
     def __init_subclass__(cls, /, default_breed, **kwargs):
         super().__init_subclass__(**kwargs)
         cls.default_breed = default_breed
 
 class Dog(Pet, default_name="German Shepherd"):
     pass

In the code above we are adding a keyword parameter to the base class that can be set when defining the subclass. This method may be used in real use cases where you want to deal with the supplied parameter and not just assign it to a property.

It seems very obscure and rarely used, but in fact you've probably encountered it many times, as it's generally used when building APIs, such as in SQLAlchemy or Flask Views.

Another magical method for metaclasses is __call__. This method allows customization of what happens when a class instance is called: the

class CallableClass:
     def __call__(self, *args, **kwargs):
         print("I was called!")
 
 instance = CallableClass()
 
 instance()
 # I was called!

It can be used to create a class that cannot be called: the

class NoInstances(type):
     def __call__(cls, *args, **kwargs):
         raise TypeError("Can't create instance of this class")
 
 class SomeClass(metaclass=NoInstances):
     @staticmethod
     def func(x):
         print('A static method')
 
 instance = SomeClass()
 # TypeError: Can't create instance of this class

For classes with only static methods, this method is used without creating an instance of the class.

Another similar scenario is the singleton pattern - a class can have at most one instance: the

class Singleton(type):
     def __init__(cls, *args, **kwargs):
         cls.__instance = None
         super().__init__(*args, **kwargs)
 
     def __call__(cls, *args, **kwargs):
         if cls.__instance is None:
             cls.__instance = super().__call__(*args, **kwargs)
             return cls.__instance
         else:
             return cls.__instance
 
 class Logger(metaclass=Singleton):
     def __init__(self):
         print("Creating global Logger instance")

The Singleton class has a private __instance - if it doesn't, it will be created and assigned a value, and if it already exists, it will just be returned.

Suppose you have a class and you want to create an instance of it without calling __init__. The __new__ method can help with this.

class Document:
     def __init__(self, text):
          = text
 
 bare_document = Document.__new__(Document)
 print(bare_document.text)
 # AttributeError: 'Document' object has no attribute 'text'
 
 setattr(bare_document, "text", "Text of the document")

In some cases, we may need to bypass the usual process of creating an instance, and the code above demonstrates how to do this. Instead of calling Document(...), we call Document.__new__(Document), which creates a bare instance without calling __init__. As a result, the attribute of the instance (text in this case) is not initialized, requiring us to additionally use the setattr function to assign a value (it's also a magic method __setattr__).

Why do it. Because we might want to replace the constructor with something like.

class Document:
     def __init__(self, text):
          = text
     
     @classmethod
     def from_file(cls, file):  # Alternative constructor
         d = cls.__new__(cls)
         # Do stuff...
         return d

The from_file method is defined here, which acts as a constructor that first creates the instance using __new__ and then configures it without calling __init__.

The next magic method related to metaprogramming is __getattr__. This method is called when a normal attribute access fails. This can be used to delegate access/calls to missing methods to another class: the

class String:
     def __init__(self, value):
         self._value = str(value)
 
     def custom_operation(self):
         pass
 
     def __getattr__(self, name):
         return getattr(self._value, name)
 
 s = String("some text")
 s.custom_operation()  # Calls String.custom_operation()
 print(())  # Calls String.__getattr__("split") and delegates to 
 # ['some', 'text']
 
 print("some text" + "more text")
 # ... works
 print(s + "more text")
 # TypeError: unsupported operand type(s) for +: 'String' and 'str'

We want to add some extra functions to the class (like custom_operation above) to define a custom implementation of string. But we don't want to re-implement every string method such as split, join, capitalize, etc. Here we can use __getattr__ to call these existing string methods.

While this applies to normal methods, note that in the above example, the magic method __add__ (which provides operations such as connections) is not delegated. So, if we want them to work as well, we have to reimplement them.

introspection

The last method related to metaprogramming is __getattribute__. It a looks very similar to the previous __getattr__, but they have a subtle difference, __getattr__ is only called when the attribute lookup fails, whereas __getattribute__ is called before the attribute lookup is attempted.

So you can use __getattribute__ to control access to an attribute, or you can create a decorator to log each attempt to access an instance attribute:.

def logger(cls):
     original_getattribute = cls.__getattribute__
 
     def getattribute(self, name):
         print(f"Getting: '{name}'")
         return original_getattribute(self, name)
 
     cls.__getattribute__ = getattribute
     return cls
 
 @logger
 class SomeClass:
     def __init__(self, attr):
          = attr
 
     def func(self):
         ...
 
 instance = SomeClass("value")
 
 # Getting: 'attr'
 ()
 # Getting: 'func'

The decorator function logger first logs the original __getattribute__ method of the class it decorates. It then replaces it with a custom method that records the name of the accessed attribute before calling the original __getattribute__ method.

Magic Attributes

So far, we've only discussed magic methods, but there are quite a few magic variables/properties in Python as well. One of them is __all__: the

# some_module/__init__.py
 __all__ = ["func", "some_var"]
 
 some_var = "data"
 some_other_var = "more data"
 
 def func():
     return "hello"
 
 # -----------
 
 from some_module import *
 
 print(some_var)
 # "data"
 print(func())
 # "hello"
 
 print(some_other_var)
 # Exception, "some_other_var" is not exported by the module

This attribute can be used to define which variables and functions are exported from the module. We created a Python module.../some_module/ separate file (__init__.py). There are 2 variables and a function defined in this file, and only 2 of them (func and some_var) are exported. If we try to import the contents of some_module in another Python program, we only get 2 of them.

Note, however, that the __all__ variable only affects * import as shown above; we can still import functions and variables using explicit names, such as import some_other_var from some_module.

Another common double underscore variable (module attribute) is __file__. This variable identifies the path to the file from which it is accessed: the

from pathlib import Path
 
 print(__file__)
 print(Path(__file__).resolve())
 # /home/.../directory/
 
 # Or the old way:
 import os
 print(((__file__)))
 # /home/.../directory/

This way we can combine __all__ and __file__ and can load all modules in one folder:.

# Directory structure:
 # .
 # |____some_dir
 #   |____module_three.py
 #   |____module_two.py
 #   |____module_one.py
 
 from pathlib import Path, PurePath
 modules = list(Path(__file__).("*.py"))
 print([PurePath(f).stem for f in modules if f.is_file() and not  == "__init__.py"])
 # ['module_one', 'module_two', 'module_three']

The last attribute that is important to me is __debug__. It can be used for debugging, but more specifically, it can be used for better control of assertions: the

# 
 def func():
     if __debug__:
         print("debugging logs")
 
     # Do stuff...
 
 func()

If we use the

python 

Running this code normally, we'll see the "debug log" printed out, but if we use the

python -O 

, the optimization flag (-O) will set __debug__ to false and remove debug messages. Thus, if you run your code with -O in a production environment, you don't have to worry about forgotten print calls during debugging, because none of them will be displayed.

Create your own magic method

Can we create our own methods and properties? Yes, you can, but you shouldn't.

Double underscore names are reserved for future extensions to the Python language and should not be used in your own code. If you decide to use such names in your code, then in the future if they are added to the Python interpreter, this will not be compatible with your code. So for these methods, we just have to remember and use them.

to this article on the use of Python magic method is introduced to this article, more related Python magic method content, please search for my previous articles or continue to browse the following related articles I hope you will support me in the future!