Classes and Objects III: Metaclasses, Abstract Base Classes and Class Decorators

 Metaclasses

Everything in python is an object including classes; if a class is an object then such class must have another class from which it is created.
Consider, an instance, f, of a user defined class Foo; we can find out the type/class of the instance, f by using the inbuilt method, type and in this case it seen that the type of f is Foo.


>>> class Foo(object):
...     pass
... 
>>> f = Foo()
>>> type(f)
<class '__main__.Foo'>
>>> 

Given that everything in python is an object including classes, we can also introspect on a class object to find out the type/class for such class.
To illustrate this, we introspect on our previous class, Foo, using the type inbuilt method.


class Foo(object):
    pass

>>> type(Foo)
<class 'type'>

In new style classes such as that defined above, the class used for creating all other class objects is the type class. This applies to user defined classes as shown above as well as in-built classes as shown below:

>>>type(dict)
<class 'type'>

Classes such as the type class that are used to create other classes are called metaclasses in python. That is all there is to metaclasses; they are classes that are used in creating other classes. Custom metaclasses are not often used in python but sometimes we want to control the way our classes are created; for example we may want to check that every method has some kind of documentation; this is where custom metaclasses come in handy.

Before explaining how metaclasses are used to customize class creation, we look in detail at how the creation of python class objects happens when a class statement is encountered during the execution of a python script.

# class definition
class Foo(object):
    def __init__(self, name):
        self.name = name

    def print_name():
        print(self.name)

The above snippet is the class definition for a simple class that every python user is familiar with but this is not the only way such class can be defined. The snippet below shows a more involved method for defining the same above class with all the syntactic sugar provided by the class keyword stripped away; the snippet provides a better understanding of what actually goes on under the covers during the execution of a python class:

class_name = "Foo"
class_parents = (object,)
class_body = """
def __init__(self, name):
    self.name = name

def print_name(self):
    print(self.name)
"""
# a new dict is used as local namespace
class_dict = {}

#the body of the class is executed using dict from above as local 
# namespace 
exec(class_body, globals(), class_dict)

# viewing the class dict reveals the name bindings from class body
>>>class_dict
{'__init__': <function __init__ at 0x10066f8c8>, 'print_name': <function blah at 0x10066fa60>}
# final step of class creation
Foo = type(class_name, class_parents, class_dict)

When a new class is defined, the body of the class is executed as a set of statements within its own namespace (its own dict). As a final step in the class creation process, the class object is then created by instantiating the type metaclass passing in the class name, base classes and dictionary as arguments. The above snippet shows how metaclasses comes into play during the class creation process but the above is not the way that classes are every defined rather they are defined with the class statements and it is here we want to control such metaclass.

The metaclass used in the class creation can be explicitly specified by setting a __metaclass__ variable or supplying the metaclass keyword argument in a class definition. In the case that none of this is supplied, the class statement examines the first entry in the tuple of the the base classes if any. If no base classes are used, the global variable __metaclass__ is searched for and if no value is found for this, python uses the default metaclass.

Armed with a basic understanding of metaclasses, we illustrate how metaclasses can be of use to python programmers.

 Metaclasses in Action

We can define custom metaclasses that can be used when creating classes. These custom metaclasses will normally inherit from type and re-implement certain methods such as __init__ and __new__.

We start with a trivial example.

Imagine that you are the chief architect for a shiny new project and you have diligently read dozens of software engineering books and style guides that have hammered on the importance of docstrings so you want to enforce the requirement that all non-private methods in the project must have docstrings; how would you enforce this requirement?

A simple and straightforward answer to this is to create a custom metaclass that will be used across the project that enforces this requirement. The snippet below though not production ready is an example of such a metaclass.


class DocMeta(type):

    def __init__(self, name, bases, attrs):
        for key, value in attrs.items():
            # skip special and private methods
            if key.startswith("__"): continue
            # skip any non-callable
            if not hasattr(value, "__call__"): continue
            # check for a doc string. a better way may be to store 
            # all methods without a docstring then throw an error showing
            # all of them rather than stopping on first encounter
            if not getattr(value, '__doc__'):
                raise TypeError("%s must have a docstring" % key)
        type.__init__(self, name, bases, attrs)

We create a type subclass, DocMeta, that overrides the type class __init__ method. The implemented __init__ method iterates through all the class attributes searching for non-private methods missing a docstring; if such is encountered an exception is thrown as shown below.

class Car(object):

    __metaclass__ = DocMeta

    def __init__(self, make, model, color):
        self.make = make
        self.model = model
        self.color = color

    def change_gear(self):
        print("Changing gear")

    def start_engine(self):
        print("Changing engine")

car = Car()
Traceback (most recent call last):
  File "abc.py", line 47, in <module>
    class Car(object):
  File "abc.py", line 42, in __init__
    raise TypeError("%s must have a docstring" % key)
TypeError: change_gear must have a docstring

Another trivial example that illustrates the use of python metaclasses is when we want to create a final class that is a class that cannot be sub-classed. Some people may feel that this is unpythonic but for illustration purposes we implement a metaclass enforcing this requirement below:

class final(type):
    def __init__(cls, name, bases, namespace):
        super(final, cls).__init__(name, bases, namespace)
        for klass in bases:
            if isinstance(klass, final):
                raise TypeError(str(klass.__name__) + " is final")

>>> class B(object):
...     __metaclass__ = final
... 
>>> class C(B):
...     pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in __init__
TypeError: B is final

In the above example, the metaclass simply performs a check ensuring that the final class is never part of the base classes for any class being created.

There is confusion sometimes over whether to override either __init__ or __new__ when defining metaclasses. The decision whether to use either of them depends on what we are trying to achieve with such custom metaclasses. If we are trying to modify the class by modifying some class attribute then we override the __new__ method but when we are looking just to carry out checks such as we have done above then we override the __init__ method of the metaclass.

 Abstract Base Classes

Sometimes, we want to enforce a contract between classes in our program. For example, we may want all classes of a given type to implement a set of methods and properties; this is accomplished by interfaces and abstract classes in statically typed languages like Java. In python we may create a base class with default methods and have all other classes inherit from them but what if we want each subclass to have its own implementation and we want to enforce this rule. We could define all the needed methods in a base class and have them raise NotImplementedError exception then the subclasses have to implement these methods if they are going to be used. However this does not still solve the problem fully. We could have subclasses that don’t implement this method and one would not know till the method call was attempted at runtime. Another issue we may experience is that of a proxy object that passes on method calls to another object. Even if such an object implements all required methods of a type via its proxied object, an isinstance test on such a proxy object for the proxied object will fail to produce the correct result.

Python’s Abstract base classes provide a simple and elegant solution to these issues mentioned above. The abstract base class functionality is provided by the abc module. This module defines a metaclass and a set of decorators that are used in the creation of abstract base classes.
When defining an abstract base class we use the ABCMeta metaclass from the abc module as the metaclass for the abstract base class and then make use of the @abstractmethod and @abstractproperty decorators to create methods and properties that must be implemented by non-abstract subclasses. If a subclass doesn’t implement any of the abstract methods or properties then it is also an abstract class and cannot be instantiated as illustrated below:

from abc import ABCMeta, abstractmethod

class Vehicle(object):
    __metaclass__ = ABCMeta

    @abstractmethod
    def change_gear(self):
        pass

    @abstractmethod
    def start_engine(self):
        pass


class Car(Vehicle):

    def __init__(self, make, model, color):
        self.make = make
        self.model = model
        self.color = color

# abstract methods not implemented
>>> car = Car("Toyota", "Avensis", "silver")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Car with abstract methods change_gear, start_engine
>>> 

Once, a class implements all abstract methods then that class becomes a concrete class and can be instantiated by a user.

from abc import ABCMeta, abstractmethod

class Vehicle(object):
    __metaclass__ = ABCMeta

    @abstractmethod
    def change_gear(self):
        pass

    @abstractmethod
    def start_engine(self):
        pass


class Car(Vehicle):

    def __init__(self, make, model, color):
        self.make = make
        self.model = model
        self.color = color

    def change_gear(self):
        print("Changing gear")

    def start_engine(self):
        print("Changing engine")

>>> car = Car("Toyota", "Avensis", "silver")
>>> print(isinstance(car, Vehicle))
True

Abstract base classes also allow existing classes to registered as part of its hierarchy but it performs no check on whether those classes implement the methods and properties that have been marked as abstract. This provides a simple solution to the second issue raised in the opening paragraph. Now we can just register a proxy class with an abstract base class and isinstance check will return the correct answer when used.

from abc import ABCMeta, abstractmethod

class Vehicle(object):
    __metaclass__ = ABCMeta

    @abstractmethod
    def change_gear(self):
        pass

    @abstractmethod
    def start_engine(self):
        pass


class Car(object):

    def __init__(self, make, model, color):
        self.make = make
        self.model = model
        self.color = color

>>> Vehicle.register(Car)
>>> car = Car("Toyota", "Avensis", "silver")
>>> print(isinstance(car, Vehicle))
True

Abstract base classes are used a lot in python library. They provide a mean to group python objects such as number types that have a relatively flat hierarchy. The collections module also contains abstract base classes for various kinds of operations involving sets, sequences and dictionaries.
Whenever we want to enforce contracts between classes in python just as interfaces do in java, abstract base classes is the way to go.

 Class Decorators

Just like functions can be decorated with other functions, classes can also be decorated in python. We decorate classes to add required functionality that maybe external to the class implementation; for example we may want to enforce the singleton pattern for a given class. Some functions implemented by class decorators can also implemented by metaclasses but class decorators sometimes make for a cleaner implementation to such functionality.

The most popular example used to illustrate the class decorators is that of a registry for class ids as they are created.


registry  =  {}

def register(cls):
    registry[cls.__clsid__] = cls
    return cls

@register
class Foo(object):
    __clsid__ = "123-456"

def bar(self):
    pass

Another example of using class decorators is for implementing the singleton pattern as shown below:

def singleton(cls):
    instances = {}

    def get_instance():
        if cls not in instances:
            instances[cls] = cls()
        return instances[cls]

    return get_instance

The decorator defined above can be used to decorate any python class forcing that class to initialize a single instance of itself throughout the life time of the execution of the program.


@singleton
class Foo(object):
    pass

>>> x = Foo()
>>> id(x)
4310648144
>>> y = Foo()
>>> id(y)
4310648144
>>> id(y) == id(x)
True
>>> 

In the above example, we initialize the Foo class twice; when we compare the ids of both objects, we see that they refer to a single object of the class. The same functionality can be achieved using a metaclass by overriding the __call__ method of the metaclass as shown below:

class Singleton(type):
    _instances = {}
    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]

class Foo(object):
    __metaclass__ = Singleton

>>> x = Foo()
>>> y = Foo()
>>> id(x)
4310648400
>>> id(y)
4310648400
>>> id(y) == id(x)
True

 Further Reading

  1. Python Essential Reference 4th Edition; David Beazley.
  2. Stack Overflow: Creating a singleton in python.
  3. Python Documentation: abc - Abstract Base Classes.
 
560
Kudos
 
560
Kudos

Now read this

The Function II: Python Function Decorators

Function decorators enable the addition of new functionality to a function without altering the function’s original functionality. Prior to reading this post, it is important that you have read and understood the first installment on... Continue →