Lately I needed to write some simulation scripts for my work that required intensive mathematical computations. At the beginning, I picked up MATLAB as it is very well known for easy matrix operations and supplies a big collection of libraries that provide “everything” for you. Apart from matrix operation, the language has little to offer, being quite inflexible. Coding in Matlab made me feel like I’m back to the days of BASIC and Pascal. Even worse when it comes down to OS operations like splitting processes for parallel execution you are left alone. Maybe I could spend a bit more time looking at such matters in Matlab, but soon I discovered three libraries; numpy, scipy and matplotlib for Python that solved my problems. More or less they are an alternative to Matlab’s libraries for use in Python.

In this post, I’ll give my approach on solving an object composition problem in Python based on the decorator pattern, that I doubt I would have been able to solve in Matlab as elegantly. The decorator pattern is defined, in the GoF book, as an alternative to sub-classing for dynamically extending functionality of objects instead of classes.

Motivation

As part of the simulations, I needed to code event detectors that were operating on some traces. Each detector works on a stream of input data, using a specific set of features extracted from the incoming stream. While, I wanted to explore and test what would be the most appropriate configuration of detectors, I needed to experiment with different setups. Avoiding getting into too much detail, I wanted to add/remove functionality to my detectors.

A few examples of detector setups would better illustrate the problem. In the simplest case, I wanted to collect some features from a data-set and work separately, logging the results independently. You can consider each rectangle an instance of a class while the arrows denote collaboration between objects. The direction is from the outer objects towards the inner ones, while dashed arrows imply that the object has access to the data.

Simple decoration of detectors

In another experiment, I wanted to exploit the correlation of events occurring concurrently in different sources. The setup is similar to the previous one, but instead of logging result independently as in the previous example, I funnel them through a correlation procedure to increase my accuracy of detection. The final aggregate result is then forwarded for logging.

Correlating detector outputs

The third picture presents a more complex setup. There are three different detectors, each requiring different features, while a specific detector requires a pair of features. Furthermore, events detected by detectors A and B are correlated. Consequently, feedback from each detector gives insight on the monitored event and increases the decision belief. A detector is placed there performs exactly this procedure by having as input the two originals. Finally, the aggregate result of the detection mechanism is logged.

A more complex composition of detectors

Consequently, a generic scheme was needed for composing objects, while keeping the composition transparent in order to enable client code access interfaces of inner objects. The first example follows exactly the definition of the decorator pattern. However, the rest examples involve more complex compositions of objects where the decoration chain is transformed into a directed acyclic graph (DAG). Considering that decoration is a dynamic alternative for inheritance at object-level the branching can be considered similar to multiple inheritance.

Why not Inheritance?

Apparently, creating a different detector class for each setup is not optimal. It requires creation of as many fat classes as the number of setups, where code reuse is restricted to copy-paste, hence, none at all. But why not inheritance? The composition looks like a hierarchy tree growing sideways on the graphs. First of all, this is semantically wrong. Inheritance denotes an isA relationship between two classes and in this case this is not true. While it could be a feasible implementation, it would be an abuse of the inheritance mechanism.

Inheritance would also require the definition of a class for each setup required so that an instance can later be created. Finally, composition of objects using inheritance is possible only at compile-time, thus, all possible combinations should be provided as classes for the execution to pick during run-time. It is preferable to delegate composition of objects during run-time, constructing only the exact number of composites that are required.

Implementation

The decorator pattern is defined as a dynamic technique for adding/removing functionality to an object instance, instead of a class. Dynamic refers to the time of behaviour modification, in our case during run-time. Below is a generic implementation of the decorator pattern in Python 2.6.

class decorator (attribute_forwarder):
  def __init__(self, *objs):
    self.__inners = []
    self.decorate(*objs)

  def decorate(self, *objs):
   self.__inners += [obj for obj in objs]

  def undecorate(self, *objs):
    if len(objs) == 0:
      self.__inners = []
    else:
      for obj in objs:
        self.__inners.remove(obj)

  def peel(self):
    if len(self.__inners == 1): return self.__inners[0]
    else: return decorator(*self.__inners)

A class that wants to decorate an instance has just to inherit from the class decorator and define the functionality it needs to provide. The class constructor takes as parameters the objects that are going to be decorated, while decorate() and undecorate() functions can modify the instance’s behaviour during its lifetime. These functions use the argument packing/unpacking feature of Python to support variable length for argument lists to remain generic. I’m not convinced where the undecorate() function might be usefull, as it eventually yields a hollow decorator instance, but I include it for the sake of completeness and symmetry.

The function that returns an undecorated version of the inner objects is peel(). In essence, it allows removing the outer layer of functionality and reference to the inner objects. You may have noticed that peel() actually returns a decorator in case of multiple inner objects. This may look like not removing the external layer, but it does actually remove the user decorator. While someone would expect it to return maybe a list of the previously decorated objects it will soon be apparent why a trivial decorator is preferred. Essentially, a trivial decorator can only forward attribute requests to its inner objects.

There is one piece missing though from the implementation; attribute forwarding from the outer towards the inner layers of the composite object. We extend the decorator class to implement this functionality.

class decorator:
  ...

  def __getattr__(self, fname):
    class callablelist(list):
      def __call__(self, *args):
        return [callable(*args) for callable in self]

    if len(self.__inners) == 1: return getattr(self.__inners[0], fname)
    else: return callablelist(
      [getattr(obj, fname) for obj in self.__inners if hasattr(obj, fname)]
    )

The original decorator pattern illustrates only chains of decorated objects. However, in this case we deal with graphs that spread and converge. The __getattr__() function is a special function in python that is called when a requested attribute of an object can not be found. In this case, the decorator forwards the attribute request to its inner classes. In case of multiple inner classes, it collects and returns a list with the requested attributes. The inner class callablelist is a subclass of the original python list, which supports the implementation of a generic, transparent decorator object that does not hide the interfaces of its inner objects. Class callablelist is a normal python list that defines the __call__ method, the handler function for calling an object in python, i.e. overloading calling operator. The call behaviour of the callable list is that it calls all its elements and returns a list with their respective return values.

Examples

So let’s see how the decorator class helps us with the aforementioned problem of object composition.

class F1:
  def __call__(self, x):
    return x > 5

class F2:
  def __call__(self, x):
    return x == 2*x-7

class F3:
  def __call__(self, x):
    return x % 2 == 0

class D1(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    retval = self.peel()(*args)
    # decorator processing here
    return retval

class D2(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    retval = self.peel()(*args)
    return reduce(lambda x,y: x^y, retval)

class D3(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    retval = self.peel()(*args)
    # decorator processing here
    return retval

class D4(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    retval = self.peel()(*args)
    return reduce(lambda x,y: x or y, retval)

class Corr(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    retval = self.peel()(*args)
    return reduce(lambda x,y: x and y, retval)

class Log(decorator):
  def __init__(self, *objs):
    decorator.__init__(self, *objs)
  def __call__(self, *args):
    print self.peel()(*args)

The code presents mock definitions of the classes mentioned in the motivation section. The core classes are the features (F1, F2 and F3) that yield some kind of binary feature based on their input. The decorated function in this case is going to be the __call__(), i.e. overloading call operator. D1 and D3 are two detectors that derive from the decorator class and simply return the input of the inner object. Apparently, being decorators, they do add functionality in their implementation that I omit for simplicity. D2 and D4 detectors are expecting input from one or more inner objects and reduce the output with some technique to a single value. The former does a XOR operation on the return values while the latter performs OR. Similarly, Corr class can be considered as another detector that reduces values using the AND operation. Finally, the Log class prints the return value of its inner objects.

You may have noticed that decorators use peel() internally to propagate the call to inner objects uniformly whether they encapsulate one or several objects. This is achieved by the __getattr__() function defined in decorator in combination to the callablelist class.

Not we can easily compose all the setups we want by assembling small units of computation. I define the three initial examples while for the first instead of logging each detector’s output individually, as the diagram suggests, I just aggregate them in a single Log class.

x = Log( D1(F1()), D1(F2()), D1(F3()), )

y = Log(
  Corr(
    D1(F1()),
    D1(F2()),
    D1(F3()),
  )
)

z = Log(
  D4(
    D1(F1()),
    D2(F1(), F2()),
  ),
  D3(F2()),
)

x(7), y(7), z(7)

The output of the program (in the case where D1 and D3 don’t change the inner retval) is:

[True, True, False]
False
[True, True]

Limitations

That is a fairly expressive object composition tool. It leads to decomposition of the problem in small tasks that can be composed and adapted during run-time. This is actually a consequence of the decorator pattern. There are some limitations, however, with this implementation. Arguably, the most important one is the fact that decorators have to inherit from the decorator class, so it’s kind of intrusive to the class. A side-effect of this is also that decorate(), undecorate() and peel() are reserved methods and should not be redefined by decorator classes for with different semantics.

Moreover, with the current implementation it is possible to create circles on object composition, that will eventually lead to a stack overflow exception because of recursion. This is easily implementable by defining a function isPart() that would search recursively through a composite object to check if an instance is part of the composite. It is adequate to assert the non existence of the instance inside the decorate() as a precondition. It is left out from the implementation for simplicity.