» Subscribe to the dbader.org YouTube Channel for more videos.
So what’s the
with statement good for? It helps simplify some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.
In turn this helps you write more expressive code and makes it easier to avoid resource leaks in your programs.
A good way to see this feature used effectively is by looking at examples in the Python standard library. A well-known example involves the
with open('hello.txt', 'w') as f: f.write('hello, world!')
Opening files using the
with statement is generally recommended because it ensures that open file descriptors are closed automatically after program execution leaves the context of the
with statement. Internally, the above code sample translates to something like this:
f = open('hello.txt', 'w') try: f.write('hello, world') finally: f.close()
You can already tell that this is quite a bit more verbose. Note that the
try...finally statement is significant. It wouldn’t be enough to just write something like this:
f = open('hello.txt', 'w') f.write('hello, world') f.close()
This implementation won’t guarantee the file is closed if there’s an exception during the
f.write() call—and therefore our program might leak a file descriptor. That’s why the
with statement is so useful. It makes acquiring and releasing resources properly a breeze.
Another good example where the
with statement is used effectively in the Python standard library is the
some_lock = threading.Lock() # Harmful: some_lock.acquire() try: # Do something... finally: some_lock.release() # Better: with some_lock: # Do something...
In both cases using a
with statement allows you to abstract away most of the resource handling logic. Instead of having to write an explicit
try...finally statement each time,
with takes care of that for us.
with statement can make code dealing with system resources more readable. It also helps avoid bugs or leaks by making it almost impossible to forget cleaning up or releasing a resource after we’re done with it.
with in Your Own Objects
Now, there’s nothing special or magical about the
open() function or the
threading.Lock class and the fact that they can be used with a
with statement. You can provide the same functionality in your own classes and functions by implementing so-called context managers.
What’s a context manager? It’s a simple “protocol” (or interface) that your object needs to follow so it can be used with the
with statement. Basically all you need to do is add
__exit__ methods to an object if you want it to function as a context manager. Python will call these two methods at the appropriate times in the resource management cycle.
Let’s take a look at what this would look like in practical terms. Here’s how a simple implementation of the
open() context manager might look like:
class ManagedFile: def __init__(self, name): self.name = name def __enter__(self): self.file = open(self.name, 'w') return self.file def __exit__(self, exc_type, exc_val, exc_tb): if self.file: self.file.close()
ManagedFile class follows the context manager protocol and now supports the
with statement, just like the original
open() example did:
>>> with ManagedFile('hello.txt') as f: ... f.write('hello, world!') ... f.write('bye now')
__enter__ when execution enters the context of the
with statement and it’s time to acquire the resource. When execution leaves the context again, Python calls
__exit__ to free up the resource.
Writing a class-based context manager isn’t the only way to support the
with statement in Python. The
contextlib utility module in the standard library provides a few more abstractions built on top of the basic context manager protocol. This can make your life a little easier if your use cases matches what’s offered by
For example, you can use the
contextlib.contextmanager decorator to define a generator-based factory function for a resource that will then automatically support the
with statement. Here’s what rewriting our
ManagedFile context manager with this technique looks like:
from contextlib import contextmanager @contextmanager def managed_file(name): try: f = open(name, 'w') yield f finally: f.close() >>> with managed_file('hello.txt') as f: ... f.write('hello, world!') ... f.write('bye now')
In this case,
managed_file() is a generator that first acquires the resource. Then it temporarily suspends its own executing and yields the resource so it can be used by the caller. When the caller leaves the
with context, the generator continues to execute so that any remaining clean up steps can happen and the resource gets released back to the system.
Both the class-based implementations and the generator-based are practically equivalent. Depending on which one you find more readable you might prefer one over the other.
A downside of the
@contextmanager-based implementation might be that it requires understanding of advanced Python concepts, like decorators and generators.
Once again, making the right choice here comes down to what you and your team are comfortable using and find the most readable.
Writing Pretty APIs With Context Managers
Context managers are quite flexible and if you use the
with statement creatively you can define convenient APIs for your modules and classes.
For example, what if the “resource” we wanted to manage was text indentation levels in some kind of report generator program? What if we could write code like this to do it:
with Indenter() as indent: indent.print('hi!') with indent: indent.print('hello') with indent: indent.print('bonjour') indent.print('hey')
This almost reads like a domain-specific language (DSL) for indenting text. Also, notice how this code enters and leaves the same context manager multiple times to change indentation levels. Running this code snippet should lead to the following output and print neatly formatted text:
hi! hello bonjour hey
How would you implement a context manager to support this functionality?
By the way, this could be a great exercise to wrap your head around how context managers work. So before you check out my implementation below you might take some time and try to implement this yourself as a learning exercise.
Ready? Here’s how we might implement this functionality using a class-based context manager:
class Indenter: def __init__(self): self.level = 0 def __enter__(self): self.level += 1 return self def __exit__(self, exc_type, exc_val, exc_tb): self.level -= 1 def print(self, text): print(' ' * self.level + text)
Another good exercise would be trying to refactor this code to be generator-based.
Things to Remember
withstatement simplifies exception handling by encapsulating standard uses of
try/finallystatements in so-called Context Managers.
- Most commonly it is used to manage the safe acquisition and release of system resources. Resources are acquired by the
withstatement and released automatically when execution leaves the
witheffectively can help you avoid resource leaks and make your code easier to read.