A dark cover card showing Python generator code alongside memory comparison: list takes 8MB, generator takes 104 bytes

What Is a Python Generator? Lazy Values, O(1) Memory, and Interview Patterns

You have a log file. It has two billion lines. You want to process each one.

Option one: read all two billion lines into a list, then iterate. Your program allocates several gigabytes of RAM. Your laptop fan spins up. Then it dies. Congratulations, you have just met MemoryError. Option two: read and process one line at a time, keeping exactly one line in memory at any moment. In Python, that is a generator.

A generator is a function that produces values one at a time, pausing after each yield until the caller asks for the next. It looks almost exactly like a regular function, except it uses yield instead of return. That one keyword changes everything about how the function runs.

One Keyword Changes Everything

Here is the smallest possible generator:

def count_up():
    yield 1
    yield 2
    yield 3

Call it and you get a generator object back. Not 1. Not a list. An object:

gen = count_up()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
print(next(gen))  # raises StopIteration

Each next() call resumes the function exactly where it left off, runs until the next yield, hands the value to the caller, and freezes. Local variables, loop counters, everything stays suspended between calls. Like a very cooperative function that just keeps putting itself on pause.

A regular function discards all that state the moment it returns. A generator holds onto it indefinitely.

You can also iterate a generator with a for loop. Python calls next() for you and stops cleanly on StopIteration:

for val in count_up():
    print(val)  # 1, then 2, then 3

Calling It Doesn't Run It

This is the part that trips everyone up at least once. A generator call returns immediately with a generator object. The body of the function has not executed yet. Not a single line of it.

def demo():
    print("A")
    yield 1
    print("B")
    yield 2
    print("C")

gen = demo()
# Nothing printed yet. The function body has not run.

next(gen)  # prints "A", returns 1, pauses before "B"
next(gen)  # prints "B", returns 2, pauses before "C"
next(gen)  # prints "C", runs off the end, raises StopIteration

The first time you see this, your mental model is "I called the function, so it ran." That model is wrong for generators, and it produces some extremely confusing bugs. Calling the function gives you a generator object. Running happens when you pull from it.

The generator's execution state lives in the generator object itself, not the call stack. Each next() call borrows a frame of execution, runs until yield, and returns it. The frame does not get popped until the generator is garbage collected or explicitly closed.

For a deeper look at how the call stack handles function frames, see recursion space complexity and what each frame actually stores.

Infinite Sequences in Five Lines

Here is the Fibonacci sequence as a list versus a generator:

# List: materializes all n values in memory
def fib_list(n):
    result = []
    a, b = 0, 1
    for _ in range(n):
        result.append(a)
        a, b = b, a + b
    return result

# Generator: computes one value at a time, stores none
def fib_gen():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

Yes, while True is intentional. The generator runs forever. You pull from it as long as you need and stop. The sequence does not care what you do with it. It just sits there, ready:

gen = fib_gen()
for _ in range(10):
    print(next(gen))  # 0 1 1 2 3 5 8 13 21 34

Infinite sequences are impossible to represent as a list. With a generator they take five lines.

O(1) Memory, No Matter the Sequence Length

Here is the same thing measured:

import sys

numbers_list = list(range(1_000_000))
print(sys.getsizeof(numbers_list))  # ~8,697,464 bytes (~8.3 MB)

numbers_gen = (x for x in range(1_000_000))
print(sys.getsizeof(numbers_gen))   # 104 bytes

Eight megabytes versus 104 bytes. For the same million numbers. The generator holds no numbers at all. It holds a recipe.

At any moment, exactly one value occupies memory, regardless of the total sequence length. The time complexity is unchanged. You still do O(n) total work to consume n values. But the footprint drops from O(n) to O(1). For large n, this is the difference between a program that runs and one that does not.

There is a real cost to track: a generator can only be iterated once. After exhaustion, it is done. Call next() on an exhausted generator and you get StopIteration immediately. If you need to iterate the same sequence twice, convert to a list first or create a new generator.

One Character, Lazy Everything

Python lets you write generators inline, similar to list comprehensions:

# List comprehension: builds the full list immediately
squares_list = [x**2 for x in range(1000)]

# Generator expression: lazy, one value at a time
squares_gen = (x**2 for x in range(1000))

The only syntactic difference is parentheses instead of square brackets. Easy to miss. Very different behavior. Pass generator expressions directly to any function that accepts an iterator:

total = sum(x**2 for x in range(1_000_000))  # no intermediate list
maximum = max(len(word) for word in open("file.txt"))

This connects directly to eager vs lazy evaluation: list comprehensions are eager (compute everything now), generator expressions are lazy (compute on demand).

Chain Them: Each Stage Sees One Value

Generators compose naturally. You can chain them into a pipeline where each stage processes one value and passes it forward. If you have used Unix pipes, this is the same idea, except in Python:

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

def filter_nonempty(lines):
    for line in lines:
        if line:
            yield line

def parse_records(lines):
    for line in lines:
        yield line.split(",")

pipeline = parse_records(filter_nonempty(read_lines("data.csv")))
for record in pipeline:
    process(record)

At any point in this pipeline, only one line is in memory. The file is never fully loaded. Each generator hands off one value at a time to the next.

yield from (Python 3.3+) makes delegation cleaner:

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

list(flatten([1, [2, [3, 4]], 5]))  # [1, 2, 3, 4, 5]

yield from is equivalent to a loop that re-yields each value from the inner generator. It also threads send() and throw() through properly, which matters for coroutines.

JavaScript Has Them Too

JavaScript generators arrived in ES6:

function* fibGen() {
    let a = 0, b = 1;
    while (true) {
        yield a;
        [a, b] = [b, a + b];
    }
}

const gen = fibGen();
console.log(gen.next()); // { value: 0, done: false }
console.log(gen.next()); // { value: 1, done: false }

The function* syntax marks a generator function. Each next() call returns {value, done}. When the generator returns (not yields) a value, done is true. JavaScript generators underpin async iterators and some streaming patterns in Node.js. The concept is identical to Python's, just with a bit more syntax to type.

Python Generator Interview Patterns

Generators appear in three types of interview problems.

Large dataset problems. Any question mentioning "stream of data," "too large to fit in memory," "process line by line," or "chunk the input" is pointing at lazy iteration. Materializing everything first is the O(n) space answer. A generator is the O(1) space answer. Interviewers notice the difference.

Infinite sequence problems. "Implement an iterator over all prime numbers." "Write a function that yields Fibonacci numbers indefinitely." "Design a round-robin scheduler." These cannot be expressed as a list. A generator is the natural answer and the only sane one.

Memory optimization. If you write a function that builds a large intermediate list before processing it, an experienced interviewer will ask "can you do this without materializing the full list?" The answer is almost always a generator. This question is essentially a test of whether you know generators exist.

Common patterns worth having ready:

# Chunking a sequence into pieces of size k
def chunks(lst, k):
    for i in range(0, len(lst), k):
        yield lst[i:i + k]

# Windowing: sliding windows of size k
def windows(lst, k):
    for i in range(len(lst) - k + 1):
        yield lst[i:i + k]

# Merging two sorted generators
def merge_sorted(g1, g2):
    a, b = next(g1, None), next(g2, None)
    while a is not None or b is not None:
        if b is None or (a is not None and a <= b):
            yield a
            a = next(g1, None)
        else:
            yield b
            b = next(g2, None)

The merge pattern shows up in K-sorted streams problems. Generators also connect to memoization in an interesting way: memoization trades space for time by caching prior results, while generators trade time for space by never storing them. Often the right trade-off pulled in opposite directions.

When Not to Use a Generator

If you need random access, generators do not support it. gen[5] will fail immediately. Lists do.

If you need to iterate the sequence more than once, you must create a new generator each time or convert to a list. An exhausted generator will not reset for you. It is done and it knows it.

If the values are cheap to compute and you will consume all of them, the overhead of repeated next() calls can make generators slower than simple list iteration in tight benchmarks.

Use a generator when you do not know how many values you will need, when the sequence is too large for memory, or when you want to express an infinite or lazily-computed sequence.

If you want to practice explaining this distinction out loud, the way you would in an actual interview, SpaceComplexity runs voice-based mock interviews that score exactly this kind of design reasoning with rubric-based feedback.