Generators and Context Managers

Overview#

Generators provide a memory-efficient way to produce sequences of values one at a time, instead of building an entire collection in memory upfront. Context managers guarantee that setup and cleanup code runs correctly around a block of code, even when errors occur. Both features rely on Python's underlying iterator protocol and are central to writing efficient, robust Python. This notebook covers generator functions, generator expressions, the yield keyword in depth, the send/throw/ close generator methods, custom iterators, the with statement, writing context managers with classes and with contextlib, and common real-world patterns for both.

1. The Iterator Protocol (Foundation)#

Before generators, it helps to understand what makes something iterable. An object is iterable if it implements __iter__, which returns an iterator — an object implementing both __iter__ (returning itself) and __next__ (producing the next value, raising StopIteration when exhausted).

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class CountUpTo:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current


for n in CountUpTo(5):
    print(n)   # 1, 2, 3, 4, 5

Writing iterators by hand like this works, but requires tracking state manually (self.current) and is verbose. Generators solve exactly this problem.

2. Generator Functions#

A generator function looks like a normal function, but uses yield instead of (or alongside) return. Calling it doesn't run the function body immediately — it returns a generator object, which produces values lazily, one at a time, only as they're requested.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def count_up_to(limit):
    current = 1
    while current <= limit:
        yield current
        current += 1


gen = count_up_to(5)
print(gen)          # <generator object count_up_to at 0x...>
print(next(gen))     # 1
print(next(gen))     # 2

for n in gen:         # continues from where it left off: 3, 4, 5
    print(n)

Each call to next() resumes the function right after the last yield, runs until the next yield (or the function ends), and returns that value. When the function finishes without yielding again, Python raises StopIteration automatically — which is exactly what for loops expect and handle for you.

Why generators matter: memory efficiency#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Builds the ENTIRE list in memory — expensive for large ranges
def squares_list(n):
    return [i ** 2 for i in range(n)]

# Produces one value at a time — constant memory usage
def squares_gen(n):
    for i in range(n):
        yield i ** 2

# Both produce the same values when iterated, but squares_gen(10_000_000)
# uses a tiny, constant amount of memory, while squares_list would need
# to allocate a list of ten million integers up front.
total = sum(squares_gen(10_000_000))

3. Generator Expressions#

A generator expression is the lazy equivalent of a list comprehension — same syntax, but with parentheses instead of square brackets.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
squares = (n ** 2 for n in range(10))     # generator, not a list
print(squares)                             # <generator object ...>
print(list(squares))                       # [0, 1, 4, 9, ..., 81]

# Directly usable in functions that accept an iterable, without ()
total = sum(n ** 2 for n in range(1000))
has_even = any(n % 2 == 0 for n in [1, 3, 5, 6])

Once a generator (or generator expression) is exhausted, it cannot be restarted — it must be recreated.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
gen = (n for n in range(3))
print(list(gen))    # [0, 1, 2]
print(list(gen))    # [] — already exhausted!

4. Multiple yield Statements and State#

A generator can yield multiple times, and it remembers its full local state — variables, loop position, everything — between each call.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def conversation():
    print("Starting")
    name = yield "What is your name?"
    print(f"Got name: {name}")
    yield f"Nice to meet you, {name}"


gen = conversation()
print(next(gen))            # Starting  /  What is your name?
print(gen.send("Alice"))    # Got name: Alice  /  Nice to meet you, Alice

gen.send(value) resumes the generator, and the sent value becomes the result of the yield expression that was paused. This two-way communication is a more advanced pattern, often used in coroutine-style code.

yield from: delegating to a sub-generator#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def inner():
    yield 1
    yield 2

def outer():
    yield from inner()
    yield 3

print(list(outer()))   # [1, 2, 3]

yield from flattens the delegation — values from inner() are yielded directly by outer(), without needing an explicit loop.

5. Generator Methods: send, throw, close#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def counter():
    try:
        while True:
            yield "counting"
    except GeneratorExit:
        print("Generator closed cleanly")

gen = counter()
next(gen)
gen.close()   # prints "Generator closed cleanly"

6. Practical Generator Patterns#

Reading large files lazily#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def read_large_file(path):
    with open(path, "r") as f:
        for line in f:
            yield line.strip()

# Only one line is in memory at a time, no matter how large the file is
for line in read_large_file("huge_log.txt"):
    if "ERROR" in line:
        print(line)

Infinite sequences#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def infinite_counter(start=0):
    n = start
    while True:
        yield n
        n += 1

counter = infinite_counter()
first_five = [next(counter) for _ in range(5)]
print(first_five)   # [0, 1, 2, 3, 4]

Chaining generators (a simple pipeline)#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def read_numbers(numbers):
    for n in numbers:
        yield n

def filter_even(numbers):
    for n in numbers:
        if n % 2 == 0:
            yield n

def square(numbers):
    for n in numbers:
        yield n ** 2

pipeline = square(filter_even(read_numbers(range(10))))
print(list(pipeline))   # [0, 4, 16, 36, 64]

Each stage processes one value at a time as it flows through, rather than materializing an intermediate list at every step.

7. The with Statement and Context Managers#

A context manager guarantees setup and teardown code runs around a block, even if an exception occurs inside it. The most familiar example is file handling.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("data.txt", "r") as f:
    contents = f.read()
# file.close() is called automatically here — even if read() raised an error

This is equivalent to, but far safer than:

Python 3.12 (WASM)
EditCopyAsk AIRun Code
f = open("data.txt", "r")
try:
    contents = f.read()
finally:
    f.close()

Multiple context managers at once#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("input.txt") as infile, open("output.txt", "w") as outfile:
    outfile.write(infile.read().upper())

8. Writing a Context Manager with a Class#

A class becomes a context manager by implementing __enter__ and __exit__.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class ManagedResource:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print(f"Acquiring {self.name}")
        return self               # becomes the value bound by "as"

    def __exit__(self, exc_type, exc_value, traceback):
        print(f"Releasing {self.name}")
        if exc_type is not None:
            print(f"Handled exception: {exc_value}")
        return False               # False (or None): don't suppress the exception


with ManagedResource("database connection") as res:
    print(f"Using {res.name}")
    # raise ValueError("oops")   # __exit__ would still run first

The three parameters to __exit__ describe any exception that occurred inside the with block: its type, the exception instance, and its traceback — all None if the block completed successfully. Returning True from __exit__ suppresses the exception; returning False (or nothing) lets it propagate normally after cleanup runs.

9. Writing a Context Manager with contextlib#

For simpler cases, @contextlib.contextmanager turns a generator function into a context manager, using yield to mark the boundary between setup and teardown.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
from contextlib import contextmanager

@contextmanager
def managed_resource(name):
    print(f"Acquiring {name}")
    try:
        yield name                # value bound by "as"; block executes here
    finally:
        print(f"Releasing {name}")   # always runs, even if the block raised


with managed_resource("database connection") as res:
    print(f"Using {res}")

Everything before yield is the __enter__ logic; everything after (in the finally) is the __exit__ logic. This is often more concise than writing a full class for simple resource-management cases.

Timing a block of code#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
from contextlib import contextmanager
import time

@contextmanager
def timer(label):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f"{label} took {elapsed:.4f}s")


with timer("data processing"):
    total = sum(n ** 2 for n in range(1_000_000))

Suppressing specific exceptions#

contextlib.suppress is a built-in context manager for cleanly ignoring expected exceptions, without a full try/except.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
from contextlib import suppress

with suppress(FileNotFoundError):
    import os
    os.remove("maybe_missing_file.txt")
# no error raised even if the file doesn't exist

10. Comparing Generators and Context Managers#

Although they solve different problems, both rely on the same underlying mechanism: a function paused at yield, resumed later. This is why contextlib.contextmanager can turn a generator into a context manager — the code before yield runs on entry, the code after runs on exit, mirroring how a generator pauses and resumes at each yield.

Notes#

• A generator function is any function containing yield anywhere in its body — Python detects this at definition time and changes how the function behaves entirely.
• Generators are iterators, but not all iterators are generators — custom classes with __iter__/__next__ are iterators without being generators.
• Once exhausted, a generator cannot be reset or reused; create a new one by calling the generator function again.
• contextlib.contextmanager functions must yield exactly once — code before the yield is setup, code after (ideally in a finally) is cleanup.
• Context managers and try/finally solve overlapping problems, but context managers are more reusable and communicate intent more clearly at the call site.

Best Practices Checklist#

• Use a generator instead of building a full list when processing large or unbounded data
• Prefer generator expressions over list comprehensions when the result will only be iterated once
• Wrap generator cleanup logic in try/finally if resources need to be released when the generator is closed early
• Use yield from when delegating to a sub-generator instead of a manual loop
• Always use with for resources that need guaranteed cleanup (files, locks, connections) instead of manual open()/close()
• Reach for @contextlib.contextmanager for simple context managers instead of a full class with __enter__/__exit__
• Return False/None from __exit__ unless you specifically intend to suppress exceptions
• Use contextlib.suppress instead of empty try/except/pass blocks for expected, ignorable exceptions