his article is the second in a two-part series that digs deep to explore the fascinating new-style Python object model, which was introduced in Python 2.2 and improved in 2.3 and 2.4. The object model and type system are very dynamic and allow quite a few interesting tricks. In part one I described the object model and type system, explored various entities, explained the life cycle of an object, and introduced some of the countless ways to modify and customize almost everything you thought immutable at runtime. In this article I will contrast meta-classes with decorators, explore the Python execution model and explain how to examine stack frames at runtime. Finally, I will demonstrate how to augment the Python language itself using these techniques. To do this I introduce a private access-checking feature that can be enforced at runtime.

Meta classes vs. Decorators
Philip Eby published a great article about Python 2.4 decorators in the May 2005 issue of Dr. Dobb’s Journal, which I recommend if you want to understand them thoroughly. In brief, decorators are callable objects that accept the function they decorate as their sole argument. When a decorated function is invoked its decorator is executed. Most decorators will do something before and/or after invoking the original function. Decorators compete with meta-classes in the same evolutionary niche. Decorators have several advantages over meta-classes:

1. Decorators can be applied to every function (meta-classes can modify only class attributes)
2. Multiple decorators can be applied to a single function (each class can have only one meta-class)

On the other hand, decorators are much more intrusive. The presence of a decorator is visible in every decorated function. Meta-classes are much stealthier, especially when applied via a base class. This may be good or bad depending on the case. Decorators shine when you want to put different decorators on different functions and when you mess with them frequently (e.g. adding and removing decorators). It is very easy (once the decorator itself is defined) to add and remove the @decorator attribute.

Meta-classes beat decorators to the punch when it comes to applying a “decorator” to a group of methods across an entire class hierarchy. Adding new classes and new methods to the hierarchy doesn’t require any “meta-effort” since the meta-class will be applied by virtue of being attached to the base class.

Aspect-oriented programming is right up the decorators/meta-classes alley. Consider an aspect that tries three times to execute a function that throws an exception before giving up. This kind of policy could be useful for functions that exchange information across the network and would like to overcome temporary disconnects or delays. Let’s see how it can be implemented using meta-classes and decorators.

def tryThrice(func):    def tripleTry(*args, **kw):        for i in xrange(0,3):            try:                return func(*args, **kw)            except:                if i == 2:                    raise       return tripleTry              

The ‘tryThrice’ function is a factory function that implements the three strikes aspect. It accepts any function and returns a function called ‘tripleTry’ that executes it up to three times. If the executed function raises an exception, ‘tripleTry’ catches it and tries again and again. It gives up only after three failures by re-raising the exception. Note this interesting technique – A factory function that defines a nested function and binds it to a supplied input function, returns the bound nested function. This is not merely the equivalent of a function pointer in C.

The test subject will be a class called Raiser with a method called ’fail’ that prints ‘sorry, failed again’ and then raises an exception. When this method is given to ‘tryThrice’ the output looks like:

sorry, failed again sorry, failed again sorry, failed again RaiserException occured. too bad

So, the goal of this exercise is to take a class called Raiser with such a ’fail’ method and apply the ‘tryThrice’ aspect to it for every invocation using two different methods: via a meta-class and via a decorator.

The following sample code shows the metaclass approach. The meta-class (‘Metaclass’) has an __init__ method that gets the same class as input parameter ‘cls’. It iterates over all its attributes (the dict parameter) and replaces each FucntionType attribute (only ‘sorry, failed again’ in the case of ‘Raiser’) with the result of ‘tryThrice’, which is the ‘tripleTry’ function bound to the original method. Hooking up the meta-class to the Raiser class is as simple as ‘__metaclass__=Metaclass’ (first line of Raiser). I could hook the meta-class through a base class—which is arguably more elegant and allows you to apply the same meta-class to multiple classes without forcing all of them to import the metaclass’s module—but I preferred the direct approach in this case.

import types class Metaclass(type):    def __init__(cls, name, bases, dict):        methods = [item for item in dict.items() if type(item[1]) is types.FunctionType]        for name, method in methods:                            setattr(cls, name, tryThrice(method))class Raiser(object):    __metaclass__=Metaclass    def fail(self):        class RaiserException(Exception):            def __str__(self):                return "RaiserException occured. too bad"                print 'sorry, failed again'        raise RaiserException()                if __name__ == '__main__':    try:        Raiser().fail()    except Exception, e:            print e

The next bit of code shows the decorator approach. All you have to do is annotate ‘fail’ with ‘@tryThrice’ to get the exact same effect.

import types class Raiser(object):            @tryThrice        def fail(self):        class RaiserException(Exception):            def __str__(self):                return "RaiserException occured. too bad"                print 'sorry, failed again'        raise RaiserException()if __name__ == '__main__':    try:        Raiser().fail()    except Exception, e:            print e    

Hardening Python
Python, being the free spirit that it is, has no real access-checking mechanism (private, public, protected, package, etc). Any variable and any function can be accessed from any piece of code, so long as you qualify it properly. Python does provide a sort of name-hiding feature for class attributes. Attributes that start with two underscores (e.g. __blabla) and end with at most one underscore (e.g. __blabla_ is ok, __blabla__ is not ok) are implicitly prefixed by an underscore and the class name. So, __blabla becomes _classname__blabla (assuming classname is really the name of the class). Code inside the class can access the attribute using the short name (__blabla), but external code will have to use the full name (_classname__blabla).

The Puritan class in the example code below declares two “private” variables and one “non-private”. Note that the dump() method can access all variables with their regular name, while the external code must qualify the attribute name with ‘_Puritan’.

class Puritan(object):    __classPrivate = 3    __notPrivate__=5    def __init__(self):        self.__instancePrivate = 4            def dump(self):            print Puritan.__classPrivate        print self.__instancePrivateif __name__=='__main__':    p = Puritan()    try:        print Puritan.__classPrivate    except AttributeError, e:        print e            try:        print p.__instancePrivate    except AttributeError, e:        print e        p.dump()    print Puritan.__notPrivate__    print Puritan._Puritan__classPrivate    print p._Puritan__instancePrivateOutput:type object 'Puritan' has no attribute '__classPrivate''Puritan' object has no attribute '__instancePrivate'34534

Another way to get to “private” attributes is through the __dict__. This name mangling technique hasn’t been popular in the Python community. The most common practice is to prefix private attributes with a single leading underscore as in _private. A single leading underscore actually means that ‘import * from m’ will not import all names (classes, functions, variables, etc) that have a single leading underscore. Anyway, all these semi-formal schemes don’t really enforce code access verification and they are easy to circumvent. The question is how important is real access verification? The answer is it’s getting more and more important for large systems.

Python, as opposed to most other dynamic languages, is being used to develop enterprise-grade systems. Bugs in enterprise-grade systems are notoriously expensive (especially if they are discovered late in the development cycle). Everything that can help reduce the number of bugs is welcome. In a large team of developers there will inevitably be someone who likes shortcuts, and will therefore call this private method temporarily, potentially ruining the integrity of the system. Another scenario where access verification may be important is when your system exposes a Python API and loads plugins written by some third party. In this case, you are potentially exposed to both clumsy and malevolent individuals. This trend of scaling up Python to ever larger systems is evident also in the quest for optional static typing for Python by Guido Van Rossum, Python’s creator, and others.

Let’s assume I convinced you Python needs code access verification. What can you do about? It turns out there is plenty you can do. You can decide to focus on renaming all the private attributes in your code and the libraries you use to the double underscore style. Then, you can review your code and make sure nobody is accessing something private. When you get tired, you can decide to write a little program that will do it for you. Finally, you can run this program periodically to scan your code for violations. This approach may turn out to be too tedious and error-prone. Also, you can’t really say much about the code using static analysis if your code contains eval(), exec(), and friends or uses various dynamic code modification tricks.

The solution I’ll present is based on access verification checks at runtime (of class attributes). Whenever a private attribute is accessed, by some mysterious magic the caller will be checked, and if it doesn’t belong to the same class an exception will be raised.

Functions, Code Objects and Frames
Before you can submerge yourself into peeking and poking the call stack let’s clear the dust out of some basic concepts. When you write a function ‘foo’ you type the arguments and the code that operates on these arguments (and possibly on the environment) and you decide whether or not foo() returns a result. When the module that contains your ‘foo’ is loaded (or compiled to .pyc) Python takes your function, compiles it to a code object that contains a bunch of metadata as well as a bytecode that can be executed by the Python virtual machine. In addition, Python creates a function object that contains a bunch of different metadata and also a reference to the code object and finally puts it in the global dictionary of the module. At runtime when function ‘foo’ executes, a frame object is created and put at the top of the call stack. This frame object has yet another set of metadata and a reference to the same code object referenced by ‘foo’.

It turns out that Python can provide a lot of information about the entire call stack and particularly the direct caller.

Listing 1 is a miniature tour de force of this confusing compile-time/run-time code management. The ‘dumpObject’ function is a helper function that accepts an object and a regular expression filter. It traverses the object’s attributes and prints the name and value (by eval()uating it) of each attribute that matches the filter. This is convenient for exploring the relevant attributes of function, code and frame objects since their attributes have a distinctive prefix (func_, co_, and f_). The ‘a’ function gets the current frame object using sys._getframe() and then it calls dumpObject three times&#151for the frame object, the code object, and itself (the ‘a’ function object)?with the corresponding regular expression filter. The output is somewhat censored. I removed the frame’s builtins attribute because it was too big and the bytecode of the code object’s co_code attribute since it was unprintable binary goo. As you can see, the frame object and the function object share the same code object (at 009AF620).

Working with Stack Frames
Finding the caller is definitely on the agenda if you want to verify it is allowed to call the current method. It turns out that Python can provide a lot of information about the entire call stack and particularly the direct caller. sys._getframe() is your beachhead to the call stack. When called without parameters it returns the current frame object of the call stack. When called with a depth argument it returns the frame object in this depth in the callstack. So, the caller frame object can be obtained using sys._getframe(1). Frame objects have several useful attributes such as the context of the current frame (f_builtins, f_locals, f_globals), the caller frame (f_back), and more. I will concentrate on f_code, which is the code object associated with a frame.

Listing 2 is focused on retrieving some information about the caller using a helper function: ‘getCallerInfo’. Class A defines a method bar(), which is called from the main() function (a.bar()) and also from the a.foo() method. The a.bar() method calls getCallerInfo() to get the information and then displays it. Note that when getCallerInfo() is called from a.bar() the actual caller is already at depth 2 in the callstack, so getCallerInfo() uses sys._getFrame(2) to get its stack frame. It retrieves the caller’s module and function name from the code object; then comes the interesting part. There is no “Pythonic” way to get the argument values from the frame or code object. Luckily the ‘inspect’ module provides a function called getargvalues() that returns a tuple whose first member is a list of the argument names and whose third member is the locals dictionary of the input frame. getCallerInfo() assumes that if the first argument exists it is the ‘self’ reference and its type is therefore the class of the caller. It is possible to retrieve the first line of the caller function and parse it to determine if it’s a regular function or a method and scroll up and find the actual class. I prefer not to engage in such elaborate acrobatics at the moment (I did something similar for C++ in my article “Method Call Interception” in the April 2005 issue of C/C++ Users Journal).

Verifying Access to Private Methods
This is where I make a big messy knot of all the stuff discussed earlier and spit some code, which I hope will be useful. Here is the plan:

• Write access verification code based on call stack frame inspection
• Define a meta-class that inject access verification code to private methods
• Define a base class that has the aforementioned meta-class
• All classes that should be checked will inherit from the base class

Listing 3 contains the CodeAccessVerifier module. The module consists of the CodeAccessViolation exception, the Metaclass [meta]class, the AccessChecker class, and the _decorate and _verfiy functions.

The CodeAccessError exception is raised when an out-of-class code tries to call a private method. It contains the frame info (method, filename, class) of the called private method and the caller.

The __init__() method of the Metaclass kicks into action when any application class that inherits from the CodeChecker class is created. It iterates over all the methods of the application class and replaces all the private ones with the decorated function returned from _decorate(). Note the commented lines that check if a certain environment variable exists and return immediately if not. This is a quick and dirty way to enable/disable the meta-functionality by nipping it in the bud.

The _decorate() function gets a naked private method and returns a nested function called decorated(), which calls _verify and then the original private method (called func in this context). Note that func can have any signature since decorated() utilizes the *args, **kw arguments that allow an arbitrary number of arguments to be passed. _decorate() pulls a nifty little trick out of its sleeve: It adds an attribute called func_class that contains the class object to the original function object (func). The reason will be clear soon.

The _verify() function is called now whenever a decorated function (originally a private method) is called, just before the original method. _verify() calls _getFrameInfo() with depth 1 and 2 to get the frame information for the decorated private method and its caller. If the caller is a method of the same class (from the same file) all is well, otherwise it raises the dreaded CodeAccessError.

The _getFrameInfo() function is pretty similar to getCallerInfo() from Listing 2. The difference is that it expects a depth argument (getCallerInfo() always extracted from depth 2) and that it is aware of the custom ‘func_class’ attribute that _decorate() injected to the function object. It tries to retrieve the func_class attribute to get the class that corresponds to frame’s method (if it is indeed a method). The original function ‘func’ is available in the locals() dictionary (args[3]). If there is no ‘func’ or no ‘func.func_class’ it tries to guess the class by assuming the first argument is the ‘self’ argument and its type is the class.

Note the subtle navigation to various objects starting with the frame object. If the ‘func’ attribute exists and it has the custom ‘func_class’ attribute then you are dealing with the wrapped function and you get all information from func.func_code (the code object associated with the wrapped function). Otherwise, it’s a caller and the information is retrieved directly from f.f_code (the code object associated with caller function).

Will you take my word that it works? Yeah, I didn’t think so. I guess I’ll have to show you. Listing 4 runs the CodeAccessVerfier through its paces. There is a class A that inherits from CodeAccessVerifier.AccessChecker. Class A has two methods: a public method add() and a private method __internalAdd(). The add() method simply calls __internalAdd() that returns the sum x+y (it’s allowed because they both belong to the same class). The main() function instantiates A and calls both methods. As you can see the call to a.add() returns properly where the call to a._A__internalAdd() raises the CodeAccessError exception.

Now do you believe me? You shouldn’t. I withheld one detail. You can cheat the code access check by writing a function in the same file as the class that expects an instance of the target class as its first argument. Such a function will be considered by CodeAccessVerifier a valid caller.

Final Pearls of Wisdom
Python has an interesting type system and execution model. It gives you a lot of leeway to push the envelope and change just about anything at runtime. You should not abuse this power. Follow the principle of least surprise. Write readable code and try to avoid invisible side effects.

It is OK for frameworks and infrastructure code to transform and inject stuff quietly under the covers, but it is generally a big no-no for application code. There is no justification to implement your business logic by dynamically contacting the Budapest office, compiling on the fly their latest base class, and deriving your class and adding a couple of new methods for good measure. You will never know what hit you when the first bug shows up.