Context Managers in Python

Controlled execution

A common pattern when working with various utilities in a program

try:
    # set things up
    # work with things
finally: 
    # tear things down after we're finished

If we do this a lot, it might be useful to have a generalization for it.

using a function:
def controlled_execution(work_with, things):
    try:
        # prepare things here
        work_with(things)
    finally:
        # tear things down here

and you'd use this pattern like so

def user_defined_function(things):
    # work with things

controlled_execution(user_defined_function, things)

As an example, let's define a controlled execution function to work with file objects. What we typically need is to open a file object, work with it and then close it. We can create a generalization for the first and last part, opening and closing file. The mid part, working with the file, usually varies from one situation to the next and would belong in the user_defined_function().

def file_controlled_execution(user_defined_function, file):
    try:
        f = open(file)
        user_defined_function(f)
    finally:
        f.close()

and now, to use it

def read_file(f):
    for l in f:
        print l

file_controlled_execution(read_file, 'phonelist.dat')

Although it's nice, it's a bit cumbersome, because in addition to the included file_controlled_execution(), we're required to create an additional function read_file() to encapsulate our user defined logic. Wouldn't it be nice to have a way to control execution of code not in a function?

One approach to go about this is to turn the controlled_execution() function into a generator and yield the object of interest:

def controlled_execution(user_defined_function):
    # set things up
    yield things
    # tear things down

to continue with our example above, let's convert our file_controlled_execution() into such a generator:

def file_controlled_execution(file):
    f = open(file)
    yield f
    f.close()

and to use it,

for f in file_controlled_execution('phonelist.dat'):
    for l in f:
        print l

This works well, but not only is it weird to have to use this syntax to do it, it's also not very Pythonic, because it makes the code less clear as to the programmer's intent.


Introducing context managers:

Python 2.5 introduced context management as an explicit generalization of the "set up/work/tear down" pattern.

A basic Context Manager class defines 2 methods: __enter__() and __exit__()

class controlled_execution:

    def __enter__(self):
        # set things up
        return thing

    def __exit__(self, type, value, traceback):
        # tear things down

using the context manager

ctx_mnger = controlled_execution()
with ctx_mnger as thing:
    # do something with thing

when the with statement is executed :

  1. python (implicitly) calls ctx_mnger.__enter__() and assigns whatever the method returns to thing.

  2. the code in the body of the with statement (do something with thing) is then executed.

  3. upon completion, ctx_mnger.__exit__() is (implicitly) called and passed the exception (if any), type, value and traceback. The __exit__() method can then look at any exception thrown, suppress it or act on it if necessary. To suppress the exception, just return a True value.

example of an __exit__() method that suppresses all TypeErrors:

def __exit__(self, type, value, traceback):
    # returns True if exception is a TypeError
    return isinstance(value, TypeError)

Let's convert our previous generator into a proper context manager:

class FileControlledExecution(object):
    def __init__(self, file):
        self.f = open(file)

    def __enter__(self):
        return self.f

    def __exit__(self, type, value, traceback):
        self.f.close()

to use it,

with FileControlledExecution('phonelist.dat') as f:
    for l in f:
        print l

The intent is clear: set things up, work with things, tear things down.

file objects as context managers

Turns out that our FileControlledExecution class became redundant in python 2.5, where the file object returned from open() became itself a context manager:

Its __enter__() method returns the object itself

f = open('phonelist.dat')
f
# <open file 'phonelist.dat', mode 'r' at 0x691c5a0>
f.__enter__()
# <open file 'phonelist.dat', mode 'r' at 0x691c5a0>

Its __exit__() method closes it

f.__exit__(None, None, None)
f
# <closed file 'phonelist.dat', mode 'r' at 0x691c5a0>

To open a file, work with it and then close it:

with open('phonelist.dat') as f:
    content = f.read()
    print content

References

http://effbot.org/zone/python-with-statement.htm

http://docs.python.org/library/contextlib.html