Python Comprehensions, Iterators & Generators Guide

Write expressive Pythonic code using list, dict, and set comprehensions alongside custom iterators, memory-efficient generators, and context managers.

Writing “Pythonic” code means leveraging Python’s unique features to make programs clean, concise, and highly expressive. One of the clearest demonstrations of the Pythonic style is comprehensions — a compact syntax for transforming and filtering collections that replaces verbose loops with a single readable line. This page also covers iterators (the protocol that makes for loops work), generators (memory-efficient value producers), and context managers (automatic resource clean-up).

Comprehensions

Comprehensions provide a concise way to build new collections from existing ones.

List Comprehensions

# Traditional loop
squares = []
for x in range(5):
    squares.append(x * x)

# Pythonic list comprehension
squares = [x * x for x in range(5)]
print(squares)   # [0, 1, 4, 9, 16]

# With a filter condition
even_squares = [x * x for x in range(10) if x % 2 == 0]
print(even_squares)   # [0, 4, 16, 36, 64]

Dictionary Comprehensions

squares_dict = {x: x * x for x in range(5)}
print(squares_dict)   # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

# Invert a mapping
original = {"a": 1, "b": 2, "c": 3}
inverted = {v: k for k, v in original.items()}
print(inverted)   # {1: 'a', 2: 'b', 3: 'c'}

Set Comprehensions

Set comprehensions automatically deduplicate results:

names = ["alice", "bob", "alice", "charlie"]
unique_lengths = {len(name) for name in names}
print(unique_lengths)   # {3, 5, 7}

Iterators

An iterator is an object that produces one value at a time. It must implement two methods:

1. __iter__() — returns the iterator object itself.
2. __next__() — returns the next value; raises StopIteration when exhausted.

class Counter:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current < self.limit:
            self.current += 1
            return self.current
        else:
            raise StopIteration

for num in Counter(3):
    print(num)   # 1, 2, 3

Every Python for loop calls iter() on the target, then repeatedly calls next() on the resulting iterator until StopIteration is raised.

Generators

Generators are a simpler way to create iterators using the yield keyword. A generator function pauses at each yield, remembers its state, and resumes on the next call — producing one value at a time rather than storing them all in memory.

Generator Functions

def simple_generator():
    yield "First"
    yield "Second"
    yield "Third"

gen = simple_generator()
print(next(gen))   # First
print(next(gen))   # Second
print(next(gen))   # Third

Generators for Large Datasets

def count_up_to(n):
    for i in range(1, n + 1):
        yield i

for num in count_up_to(5):
    print(num)   # 1 2 3 4 5

Generator Expressions

Wrapped in () instead of [], generator expressions compute values lazily — only when requested:

# List comprehension — builds the full list in memory immediately
list_comp = [x * x for x in range(1_000_000)]

# Generator expression — computes each value on demand
gen_exp = (x * x for x in range(1_000_000))

print(next(gen_exp))   # 0
print(next(gen_exp))   # 1

Use generator expressions when working with large datasets or streams where you only need one element at a time. Passing a generator directly to sum(), max(), or min() is both memory-efficient and readable:

total = sum(x * x for x in range(1_000_000))

Generators vs. Lists

Lists	Generators
Stores all values in memory	Produces one value at a time
Higher memory usage	Memory-efficient
Created with [...]	Created with yield or (...)
Supports indexing	Forward-only iteration
Better for small datasets	Better for large or infinite streams

Context Managers

Context managers manage resources by guaranteeing that set-up and clean-up code always runs, even if an exception occurs.

The with Statement

The most familiar context manager opens files safely:

with open("data.txt", "w") as file:
    file.write("Pythonic Programming")
# File is automatically closed when the block exits

Creating Custom Context Managers

Use the @contextmanager decorator from contextlib for a concise generator-based approach:

from contextlib import contextmanager

@contextmanager
def managed_resource():
    print("Acquiring resource...")
    try:
        yield "Active Resource"   # passes control to the 'with' block
    finally:
        print("Releasing resource...")

with managed_resource() as res:
    print(f"Using: {res}")

# Acquiring resource...
# Using: Active Resource
# Releasing resource...

Code before yield runs at the start of the with block. Code after yield (or in finally) runs when the block exits — whether normally or due to an exception.