Translation Note: This is a hot post on Stack overflow. The questioner claims to have grasped various concepts about Python OOP programming, but has always found metaclasses difficult to understand. He knew it must be related to introspection, but still didn't find it very understandable, and hoped that people could give some real-world examples and code snippets to help understand it, as well as in what situations metaprogramming is needed. So e-satis gave a godlike reply, which garnered 985 approval points, and some people even commented that this reply should be added to the official Python documentation. And e-satis himself got 64271 reputation points. Here's the reply (hint: it's very long)
Classes are also objects
Before you can understand metaclasses, you need to get a handle on classes in Python.The concept of classes in Python is borrowed from Smalltalk, which seems a bit peculiar. In most programming languages, a class is a set of code segments that describe how to generate an object. This still holds true in Python:
>>> class ObjectCreator(object):
… pass
…
>>> my_object = ObjectCreator()
>>> print my_object
<__main__.ObjectCreator object at 0x8974f2c>
But classes in Python are much more than that. A class is also an object. Yes, that's right, an object. As soon as you use the keyword class, the Python interpreter creates an object when it executes it. The following code snippet:
>>> class ObjectCreator(object):
… pass
…
An object will be created in memory with the name ObjectCreator. this object (class) itself has the ability to create objects (instances of the class), and that's why it's a class. However, it is still an object by nature, and so you can do the following with it:
1. You can assign it to a variable
2. You can copy it
3. You can add properties to it
4. You can pass it as a function parameter
Here are examples:
>>> print ObjectCreator # You can print a class because it's actually an object as well
<class '__main__.ObjectCreator'>
>>> def echo(o):
… print o
…
>>> echo(ObjectCreator) # You can pass the class as an argument to the function
<class '__main__.ObjectCreator'>
>>> print hasattr(ObjectCreator, 'new_attribute')
Fasle
>>> ObjectCreator.new_attribute = 'foo' # You can add attributes to the class
>>> print hasattr(ObjectCreator, 'new_attribute')
True
>>> print ObjectCreator.new_attribute
foo
>>> ObjectCreatorMirror = ObjectCreator # You can assign the class to a variable
>>> print ObjectCreatorMirror()
<__main__.ObjectCreator object at 0x8997b4c>
Creating classes dynamically
Because classes are also objects, you can create them dynamically at runtime, just like any other object. First, you can create classes in functions, just use the class keyword.
>>> def choose_class(name):
… if name == 'foo':
… class Foo(object):
… pass
... return Foo # returns the class, not an instance of the class
… else:
… class Bar(object):
… pass
… return Bar
…
>>> MyClass = choose_class('foo')
>>> print MyClass # The function returns the class, not an instance of it.
<class '__main__'.Foo>
>>> print MyClass() # You can create class instances, i.e. objects, from this class
<__main__.Foo object at 0x89c6d4c>
But that's not dynamic enough, because you still need to code the entire class yourself. Since classes are also objects, they have to be generated by something to be right. When you use the class keyword, the Python interpreter automatically creates this object. But like most things in Python, Python still provides you with ways to handle it manually. Remember the built-in function type? This old but powerful function lets you know what the type of an object is, like this:
>>> print type(1)
<type 'int'>
>>> print type("1")
<type 'str'>
>>> print type(ObjectCreator)
<type 'type'>
>>> print type(ObjectCreator())
<class '__main__.ObjectCreator'>
Here, type has a completely different ability to create classes dynamically as well. type can take a class description as an argument and return a class. (I know it's silly to have two completely different uses of the same function depending on the parameters passed in, but this is done in Python to maintain backwards compatibility)
TYPE can work like this:
type(class name, tuple of parent classes (can be null for inheritance cases), dictionary containing attributes (names and values))
For example, the following code:
>>> class MyShinyClass(object):
… pass
It can be created manually like this:
>>> MyShinyClass = type('MyShinyClass', (), {}) # Returns a class object
>>> print MyShinyClass
<class '__main__.MyShinyClass'>
>>> print MyShinyClass() # Create an instance of this class
<__main__.MyShinyClass object at 0x8997cec>
You'll notice that we're using "MyShinyClass" as the class name, and you can also use it as a variable to reference the class. Classes and variables are different, and there's no reason to complicate things here.
type accepts a dictionary to define attributes for the class, so the
>>> class Foo(object):
… bar = True
It can be translated as:
>>> Foo = type('Foo', (), {'bar':True})
And you can use Foo as if it were a regular class:
>>> print Foo
<class '__main__.Foo'>
True
>>> f = Foo()
>>> print f
<__main__.Foo object at 0x8a9b84c>
True
Of course, you can inherit from this class, so the following code:
>>> class FooChild(Foo):
… pass
It can be written as:
>>> FooChild = type('FooChild', (Foo,),{})
>>> print FooChild
<class '__main__.FooChild'>
>>> print # The bar attribute is inherited from Foo
True
Eventually you will want to add methods to your class. Simply define a function with the proper signature and assign it as a property.
>>> def echo_bar(self):
…
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
False
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True
As you can see, in Python, classes are also objects and you can create classes dynamically. This is what Python does behind the scenes when you use the keyword class, and this is done through metaclasses.
What exactly is a metaclass
A metaclass is the "thing" that creates a class. You create classes to create objects that are instances of classes, don't you? But we've already learned that classes in Python are also objects. Well, metaclasses are used to create those classes (objects), and a metaclass is a class of a class, if you want to think of it that way:
MyClass = MetaClass()
MyObject = MyClass()
You've seen what TYPE allows you to do like this:
MyClass = type('MyClass', (), {})
This is because the function type is actually a metaclass. type is the metaclass that Python uses behind the scenes to create all classes. Now you're wondering why then type is all lowercase instead of Type? Well, I guess it's for consistency with str, which is the class used to create string objects, and int, which is the class used to create integer objects. type is the class used to create class objects. You can see this by examining the class attribute. everything in Python, and by that I mean everything - is an object. This includes integers, strings, functions, and classes. They are all objects, and they are all created from a class.
>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>>foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>
Now, what about the class attribute for any class?
>>> a.__class__.__class__
<type 'type'>
>>> age.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>
So, a metaclass is something that creates an object like a class. You can call metaclasses "class factories" if you like (not to be confused with factory classes :D) type is Python's built-in metaclass, but of course, you can create your own metaclasses.
metaclass attribute
You can add the metaclass attribute to a class when you write it.
class Foo(object):
__metaclass__ = something…
[…]
If you do, Python creates the class Foo with the metaclass. be careful, there's some trickery involved. You first write class Foo(object), but the class object Foo hasn't been created in memory yet.Python looks for the metaclass attribute in the definition of the class.If it finds it, Python uses it to create the class Foo.If it doesn't find it, it creates the class with the built-in type. Read the following passage several times over. When you write the following code :.
class Foo(Bar):
pass
Python does the following:
Is there a metaclass property in Foo? If yes, Python creates a class object in memory with the name Foo by metaclass (I said class object, follow my train of thought closely). If Python doesn't find a metaclass, it will continue to look for the metaclass attribute in the Bar (parent class) and try to do the same as before. If Python can't find metaclass in any parent class, it looks for metaclass in the module hierarchy and tries to do the same. If it still can't find a metaclass, Python creates the class object using the built-in type.
Now the question is, what code can you put in a metaclass? The answer is: something that creates a class. So what can be used to create a class? type, or anything that uses type or subclasses type.
Custom metaclasses
The main purpose of metaclasses is to be able to automatically change classes when they are created. Typically, you would do this for the API, where you want to be able to create classes that match the current context. Imagine a silly example where you decide that all the attributes of the classes in your module should be in uppercase. There are several ways to do this, but one of them is by setting up a metaclass at the module level, and with this approach, all the classes in the module would be created with this metaclass, and we would just tell the metaclass to change all the attributes to uppercase and we'd be good to go.
Fortunately, metaclass can actually be called arbitrarily, it doesn't need to be a formal class (I know, certain things with 'class' in their name don't need to be a class, it helps to draw a picture to understand). So let's start here with a simple function as an example.
# The metaclass will automatically pass the arguments you would normally pass to 'type' as its own arguments
def upper_attr(future_class_name, future_class_parents, future_class_attr):
'''Returns a class object, converting all attributes to uppercase.'''
# Select all attributes that do not start with '__'
attrs = ((name, value) for name, value in future_class_attr.items() if not ('__'))
# Convert them to uppercase
uppercase_attr = dict(((), value) for name, value in attrs)
# Doing class object creation via 'type'
return type(future_class_name, future_class_parents, uppercase_attr)
__metaclass__ = upper_attr # This will apply to all classes in this module.
class Foo(object):
# We could also just define __metaclass__ here, so that it would only act on this class
bar = 'bip'
print hasattr(Foo, 'bar')
# Output: False
print hasattr(Foo, 'BAR')
# Output:True
f = Foo()
print
# Output: 'bip'
Now let's do it again, this time using a real class as a metaclass.
# Keep in mind that 'type' is actually a class, just like 'str' and 'int'
# So you can inherit from type #
class UpperAttrMetaClass(type):
# __new__ is a special method called before __init__.
# __new__ is the method used to create an object and return it.
# And __init__ is just used to initialize the passed parameters to the object
# You rarely use __new__ unless you want to be able to control the creation of objects
# Here, the object created is a class, and we want to be able to customize it, so we'll rewrite __new__ here.
# You can also do something in __init__ if you want to #
# There are also advanced uses that involve rewriting __call__ special methods, but we don't use them here.
def __new__(upperattr_metaclass, future_class_name, future_class_parents, future_class_attr):
attrs = ((name, value) for name, value in future_class_attr.items() if not ('__'))
uppercase_attr = dict(((), value) for name, value in attrs)
return type(future_class_name, future_class_parents, uppercase_attr)
But this way is not really OOP. we are calling type directly and we are not rewriting the new method of the parent class. Now let's approach it like this.
class UpperAttrMetaclass(type):
def __new__(upperattr_metaclass, future_class_name, future_class_parents, future_class_attr):
attrs = ((name, value) for name, value in future_class_attr.items() if not ('__'))
uppercase_attr = dict(((), value) for name, value in attrs)
# Reuse type.__new__ method
# It's basic OOP programming, there's no magic #
return type.__new__(upperattr_metaclass, future_class_name, future_class_parents, uppercase_attr)
You may have noticed that there is an extra parameter upperattr_metaclass, which is nothing special. The first parameter of a class method always indicates the current instance, just like the self parameter in a normal class method. Of course, I've made the name longer here for clarity. But just like self, all parameters have their traditional names. So in real product code a metaclass should look like this:
class UpperAttrMetaclass(type):
def __new__(cls, name, bases, dct):
attrs = ((name, value) for name, value in () if not ('__')
uppercase_attr = dict(((), value) for name, value in attrs)
return type.__new__(cls, name, bases, uppercase_attr)
We could also make it a bit clearer if we used the super method, which would ease inheritance (yes, you can have metaclasses, inherit from metaclasses, inherit from type)
class UpperAttrMetaclass(type):
def __new__(cls, name, bases, dct):
attrs = ((name, value) for name, value in () if not ('__'))
uppercase_attr = dict(((), value) for name, value in attrs)
return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)
That's it, there's really nothing else to say about metaclasses beyond that. The reason for the complexity of code that uses metaclasses is not because of the metaclasses themselves, but because you often use metaclasses for obscure things, relying on introspection, controlling inheritance, and so on. It's true that metaclasses are particularly useful for doing "dark magic", and thus for making things complicated. But as far as metaclasses are concerned, they are actually quite simple:
1. Creation of interceptor classes
2. Modify the class
3. Return to the modified class
Why use a metaclass class instead of a function?
Since a metaclass can accept any callable object, why use a class when it's obviously more complicated? There are several reasons:
1. The intent will be clearer. When you read UpperAttrMetaclass(type), you know what's coming next.
2. You can use OOP programming. Metaclasses can inherit from metaclasses, rewriting the methods of the parent class. Metaclasses can even use metaclasses.
3. You can organize your code better. When you use metaclasses it's certainly not for simple scenarios like the one I cited above, it's usually for more complex problems. Summarizing multiple methods into a single class is helpful and makes the code easier to read.
4. You can use special methods like new, init, and call. They can help you handle different tasks. Even though you can usually get rid of everything in new, some people feel more comfortable using init.
5. wow, the name of this thing is metaclass, surely it's not good, I have to be careful!
Why exactly do you use metaclasses?
Now back to the main topic, why on earth would you use such an error-prone and obscure feature? Well, generally speaking, you don't use it at all:
The metaclass is the magic of depth, and 99% of users shouldn't have to worry about it at all. If you're trying to figure out whether or not you need to use a metaclass, then you don't need it. Those who actually use metaclasses know exactly what they need to do, and don't need to explain why they use them at all. -- Tim Peters, leader of the Python community.
The main use of metaclasses is to create APIs. a typical example is the Django ORM. which allows you to define. like this:
class Person():
name = (max_length=30)
age = ()
But if you do it like this:
guy = Person(name='bob', age='35')
This doesn't return an IntegerField object, it returns an int, which can even be used to retrieve data directly from the database. This is possible because the metaclass is defined and some magic is used to turn the simple Person class you just defined into a complex hook to the database. the Django framework takes this seemingly complex stuff and simplifies it by exposing a simple API that uses the metaclass to recreate the code that does the The real work.
concluding remarks
First, you learned that classes are actually objects that create instances of classes. Well, in fact, classes themselves are instances, but of course, they are instances of metaclasses.
>>>class Foo(object): pass
>>> id(Foo)
142630324
Everything in Python is an object, they're either instances of classes or instances of metaclasses, except for type. type is actually its own metaclass, which isn't something you can do in a pure Python environment, it's something you can do by pulling a few tricks at the implementation level. Second, metaclasses are complex. For very simple classes, you may not want to make changes to the class by using metaclasses. You can modify classes through two other techniques:
Monkey patching
class decorators
When you need to modify classes dynamically, 99% of the time you're better off using both of these techniques. Of course, 99% of the time you don't need to dynamically modify classes at all.