A Tough Design Decision

by Marty Alchin on January 21, 2011 about Biwako and Python

When I set out to work on Biwako, I expected to have to make some hard choices, but I didn’t expect to hit one so quickly. What started as a simple feature turned out to take several days of research, trial and error before finally settling on a solution that I didn’t see coming. And in the spirit of building this framework out in the open, I’d like to spend some time sharing my experiences, in hopes of helping someone else who might face a similar problem someday.

Yesterday, I explained how Python allows a class declaration to take keyword arguments alongside the usual base class. For Biwako, I planned to use that to use that feature as a way to keep things DRY. Some field types have options that are the same for all such fields within the same file format. It doesn’t make sense to have to specify the same option over and over again, when it can instead be supplied at the class level.

So the goal is to provide arguments to the class that will then be passed into each of the fields that needs it. That puts it in a different category than, for instance, Django’s Meta options. Sure, some of those are accessed by fields, but they describe the model class itself. For Biwako, the options are much more tightly integrated with the fields themselves, which poses something of a problem.

The naïve approach

To illustrate why it’s a problem, let’s start with what I’ll call the naïve approach. The most straightforward solution I could think of, just to try to get the thing working. Basically, let Python process the Structure class declaration the way it normally would, then use the attach_to_class() method to retrieve the extra arguments from the class.

First, a quick moment to explain attach_to_class() if you’re not familiar with it. Once Python has executed the body of the Structure class declartion, the metaclass for Structure will loop through all the class attributes, looking for any that have an attach_to_class() method. When one is found, the metaclass calls the method and passes it the class object and the name that the field was assigned to.

Without this step, fields have no way to know what name they were given, so it’s a necessity in any declarative framework like this. The last time I covered declarative syntax, I only used this step to set the name, but this is also the first opportunity the field has to see the class it was assigned to, so it’s advantageous to pass that in as well. So to start, the attach_to_class() method looks something like this:

class Field:

    # ...

    def attach_to_class(self, cls, name):
        self.name = name
        label = self.label or name.replace('_', ' ')
        self.label = label.title()
        cls._fields.append(self)

So here it’s dealing with the name and assigning itself to a list of known fields. But this gets called from the __init__() method of the metaclass, which, as I explained yesterday, has all the class keyword arguments available to it. So if I pass those into attach_to_class() as well, the field can pick out which options it knows about and use them however it needs to. So for integers that take an endianness argument:

class Integer(Field):
    def __init__(self, *args, endianness=BigEndian, **kwargs):
        super(Integer, self).__init__(*args, **kwargs)
        self.endianness = endianness(self.size)

    # ...

    def attach_to_class(self, cls, name, endianness=BigEndian, **options):
        self.endianness = endianness(self.size)
        super(Integer, self).attach_to_class(cls, name, **options)

But wait, why do __init__() and attach_to_class() both take an endianness argument? Well, __init__() always has to accept it in case you want to specify it explicitly, for those rare formats where endianness might not be uniform throughout the file. So by adding options to attach_to_class() without updating __init__(), what happens now is inconsistent. If you supply endianness to just the class, it will work as you expect, but if you put it in the field instead, you get this:

  1. The field instantiates with the explicit endianness value and stores it away.
  2. The class comes along and overrides it with its default value, because attach_to_class() comes later.

And if you happen to supply it in both places, it goes something like this:

  1. The field stores away its endianness value correctly.
  2. The class comes along and overrides it with the value it was given.

So really, the field value is getting the shaft no matter what. The only time it works the right way is when you don’t pass anything into the field at all. So in order to make it work properly, attach_to_class() needs to figure out whether the argument was passed into the field or not. If it was, the class-level argument should be ignored, regardless of whether something was passed in explicitly.

Of course, the simplest way to do that is to take the default value out of __init__() and use None instead. That way, attach_to_class() can check if the attribute is set to None or something more specific. Only in the former case should it bother supplying its own value.

class Integer(Field):
    def __init__(self, *args, endianness=None, **kwargs):
        self.endianness = endianness
        super(Integer, self).__init__(*args, **kwargs)

    # ...

    def attach_to_class(self, cls, name, endianness=BigEndian, **options):
        if self.endianness is None:
            self.endianness = endianness
        # Still need to initialize it
        self.endianness = self.endianness(self.size)
        super(Integer, self).attach_to_class(cls, name, **options)

Now, believe it or not, this works. It correctly handles all four combinations of where arguments could be passed in. But it has two fairly severe problems:

These problems were enough to make me quickly realize that I needed a better solution. I figured I’d have a few options, so I wanted to spend the time to find out what I could do and how it would work. I started where you probably expected me to start. More metaclasses.

The declarative approach

You’ve probably noticed by now that I’m a big fan of declarative classes, so it shouldn’t be too surprising that my first instinct was to make another one. I figured that the arguments themselves could be pulled out of the methods and assigned as attributes to the Field class, just like fields are assigned to Structure classes. So Integer would look something like this:

class Integer(Field):
    endianness = Argument(default=BigEndian)

    # ...

Behind the scenes, though, things would work a lot like the naïve approach. __init__() would be able to tell if an argument was pass in and attach_to_class() would figure out if it needed to override the attribute or not. The only real difference is that the arguments would come from the class declaration, which saves users the trouble of seeing the whole __init__()/attach_to_class() mess.

That’s certainly prettier, and it uses a syntax I’m already expecting my users to be familiar with, so it seemed promising. But there’s another half of the problem: initializing fields on their own, outside of a Structure. As it stands, I’d still be left with undefined attributes if I wanted to use the defaults.

So I then took advantage of the fact that class attributes, such as the Argument object in the above example, can be used as descriptors. That way, the field can figure out when the testing code is trying to access it. If it doesn’t have an explicit value yet, the descriptor can provide a default, which was naturally already passed into it as an argument (to … Argument … yeah, its arguments all the way down, stay with me).

class Argument:
    def __init__(self, default=None):
        self.default = default

    # ...

    def __get__(self, instance, owner):
        if self.name not in instance.__dict__:
            # Default value to the rescue!
            instance.__dict__[self.name] = self.default
        return instance.__dict__[self.name]

So far, so good. If the field gets instantiated outside a class, it doesn’t know that at first, but as soon as one of its arguments is accessed, it quickly sets a default value if it needs to and allows the field to act as if it knew all along how it was supposed to work.

You’ll notice one key feature went missing, though: the argument can’t be initialized. The way it stands, if I were to pass in endianness=BigEndian, the endianness value would end up being the BigEndian class, rather than an instance of that class that’s been tailored to the field’s size. So we need a way to specify an initialization function for each argument as well.

For that, I turned to a decorator. In the class declaration, the endianness argument gets instantiated as an object right away. I can then add a method on that object to act as a decorator, which will allow users to mark a field method as being the initialization function for the argument.

class Argument:
    def __init__(self, default=None):
        self.default = default
        # A default initializer that does nothing
        self.initialize = lambda obj, value: value

    # ...

    def __get__(self, instance, owner):
        if self.name not in instance.__dict__:
            # Default value to the rescue!
            value = self.initialize(instance, self.default)
            instance.__dict__[self.name] = value
        return instance.__dict__[self.name]

    def init(self, func):
        self.initialize = func
        return func

# ...

class Integer(Field):
    endianness = Argument(default=BigEndian)

    @endianness.init
    def initialize_endianness(self, endianness):
        return endianness(self.size)

    # ...

Of course, there are other places where this new initialize() function would get called, but you get the idea. Now we have a way to populate arguments from two different places, give them default values, and initialize whatever values are being used. It’s certainly a prettier, friendlier approach and it works great with tests. But it has its own caveats:

It’s easy to ignore that third point, and just figure that if something works, it must be the right tool for the job. In this case, it’s an extremely heavy-handed approach to an otherwise simple problem, and with my particular reliance on the declarative sytle, I really wanted to push myself to use something else.

The placeholder

So while I was working on the declarative approach, I thought of another possibilty. The real problem with the naïve approach is that it relied on a value of None to determine if an argument had been passed into the field or not, once attach_to_class() comes along. So what if I could use a different value instead? One that not only served the same purpose, but could also be used to manage a default value?

Basically, I’d use the field’s __init__() method like normal, but instead of passing in the argument’s default value directly, it could be wrapped up in a Default object (I actually called it Arg but I like Default better). Then attach_to_class() check for an instance of that object instead, and if found, grab the default value from the object itself and use that.

class Default:
    def __init__(self, value):
        self.default_value

class Field:
    # ...

    def attach_to_class(self, cls, name, **options):
        self.name = name
        label = self.label or name.replace('_', ' ')
        self.label = label.title()
        cls._fields.append(self)
        for name, value in options:
            if hasattr(self, name) and \
              isinstance(getattr(self, name), Default):
                setattr(self, name, value.default_value)

class Integer(Field):
    def __init__(self, *args, endianness=Default(BigEndian), **kwargs):
        self.endianness = endianness
        super(Integer, self).__init__(*args, **kwargs)

The hasattr() test is necessary because this will get all the options, regardless of whether any of them actually mean something to this particular field.

This approach works for the most part, but again the initialization bit has gone missing. Since the Default object gets created inside the function signatures, there’s no good place to put the initialization code. There are three options that I can see:

None of those are very convenient for users implementing their own fields. Technically a fourth could be initializing the value in attach_to_class(), but then we’re back to having problems with testing.

All in all, I didn’t get very far into this one before abandoning it due to lack of flexibility for things like initialization. Providing a default value was easy enough, but it wasn’t long before things got considerably more hairly. So when I went back to the drawing board, I thought much simpler.

Double initialization

A thought had occured to me: what I’m really trying to do is initialize the arguments in two different places. The first happens when __init__() is called during the creation of the field object. The second happens inside of attach_to_class(). But ultimately it’s the same process either way, so why not just call __init__() twice?

This one works by storing away the arguments that were passed in when the field was created, then filling in class-level arguments wherever a field-level argument wasn’t passed in. So in order to avoid messing with __init__() on the field any more than I had to, I built a very small metaclass for fields to work with—much smaller than the declarative approach shown earlier.

class FieldMeta(type):
    def __call__(cls, *args, **kwargs):
        # This gets called before __new__() or __init__()
        field = super(FieldMeta, cls).__call__(*args, **kwargs)
        field._arguments = (args, kwargs)
        return field

class Field(metaclass=FieldMeta):
    def attach_to_class(self, cls, name, **options):
        args, kwargs = self._arguments
        options.update(kwargs)
        self.__init__(*args, **options)

class Integer(Field):
    def __init__(self, *args, endianness=BigEndian, **kwargs):
        super(Integer, self).__init__(*args, **kwargs)
        self.endianness = endianness(self.size)

So the way this works, when you create a field, it’s complete, right out of the box. It has the necessary default values, initialization happens in __init__() where it belongs, there are no pesky placeholders or strange syntax and it works just as well for testing as it does in real use. It’s a fairly clean, concise solution … except for one potential problem.

Ordinarily, __init__() is called exactly once for a given object. When it’s first created, __init__() gets a chance to set some of its values to their starting positions, making the object ready for general use. Doing this step twice won’t cause problems in most cases, especially since the field won’t really get used between the two calls to __init__().

But what actually goes on in __init__() is entirely up to you. You might just set a few attributes and be done with it or you might add the field to some internal registry for later use. That latter option would be considered a side-effect: the code modifies something outside of its own scope, and that change persists even after the function is done executing.

If the code gets run twice, any side-effects would also occur twice, which could be a problem. In the case of a field registry, you would end up with each field occurring twice in the registry, which would definitely cause problems. And since there’s no obvious cue that __init__() would get called twice, such problems could be difficult to track down to their source.

What I really need is a way to just get access to the class-level arguments before the fields are created, so that I can just pass in the full, correct set of arguments the first time and save myself all this mess.

Thread locals

Now, if you’re familiar with programming, and if you’ve been following Django design discussions in particular, you might see the phrase “thread locals” and immediately jump to scenes of mass descruction, flesh being torn off from innocent bystanders and babies having their candy taken away. But I felt it was my responsibility to consider every option.

The foundation of this approach is actually a newer feature of metaclasses that I neglected to mention yesterday: the __prepare__() method. In Python 3, metaclasses can have a method called __prepare__(), which will get called before Python processes any of the contents of the class declaration. That is, before any of the fields have been created or initialized. Thankfully for us, __prepare__() also gets the same keyword argument dictionary as __new__() and __init__(), containing all the options that are declared at the top of the class.

With that information available so early on, it’s possible to store those options right away. But we need a good place to put them. Unfortunately, because __prepare__() gets called so early, it doesn’t have access to the class object yet (that hasn’t even been created yet). Instead, it gets the name of the class, a tuple of its base classes and the dictionary of keyword arguments. So we turn to thread locals.

In a nutshell, thread locals are a way to store data so that only the current thread can see it. That way, if more than one thread happens to be running the same code at about the same time, there won’t be any conflict between the two. Since a single thread can only run code sequentially, we can be sure that if we place those class-level options in thread-local storage, it’ll be available exactly when we need it, and only to the thread that should see it.

Then, when each field is called, it can look in thread-local storage to find any class-level arguments and combine them with its own arguments before calling __init__() in the first place.

import threading

class FieldMeta(type):
    _registry = threading.local()
    _registry.options = {}

    def __call__(cls, *args, **kwargs):
        if FieldMeta._registry.options:
            options = FieldMeta._registry.options.copy()
            options.update(kwargs)
        else:
            options = kwargs
        return super(FieldMeta, cls).__call__(*args, **options)

class Field(metaclass=FieldMeta):
    # ...

class Integer(Field):
    def __init__(self, *args, endianness=BigEndian, **kwargs):
        super(Integer, self).__init__(*args, **kwargs)
        self.endianness = endianness(self.size)

class StructureMeta(type):
    @classmethod
    def __prepare__(metacls, name, bases, **options):
        FieldMeta.options.arguments = options

        # ...

    def __init__(cls, name, bases, attrs, **options):
        # ...

        # Clean up the thread-local dictionary
        FieldMeta.options.arguments = {}

With this in place, each field has the opportunity to instantiate itself using all of the arguments that it needs, regardless of where they were specified, without requiring any special handling for the subclass. Initialization of arguments happens in __init__(), and that process happens exactly once per field, just like it should. Fields get their default values when creating them for testing, and they get their full options in real use.

The decision

So after going through this entire process (over about five days), I finally had to come to a decision. As much as I had expected to find something more traditional, it turned out that thread locals actually solved the problem more cleanly than any other. It made the API dead obvious by not changing anything, and it was actually pretty simple to implement.

So there you have it. Biwako uses thread locals, and now you know why.

The moral of the story

Now, this hasn’t been a story of thread locals, metaclasses, descriptors or anthing else. Really, I just wanted to show the design process. Writing a framework requires countless decisions like this, and they can sound really daunting.

Sometimes it can seem like framework authors just naturally have things they like and don’t like or that they somehow always have all the right answers. But the fact is, most of these decisions are the result of a lot of trial and error, soul searching and trying very hard to think of the user experience above all else. Design and usability doesn’t only have to be visual; API design uses many of the same principles.

So Think of your users. Research your options. Keep an open mind. Ride the wave and you might be surprised where the process takes you.