Lesson 2.3

Iterators and generators

A generator produces values one at a time as they are asked for.

15mBeginner22.6k students

Overview

Streaming instead of holding

A generator produces values one at a time as they are asked for. Reading a multi-gigabyte log becomes possible on a laptop, because only the current line is ever in memory.

Writing one is a matter of yielding instead of appending to a list and returning it. Generators also compose: piping one into another builds a processing chain that still streams end to end.

The catch is that a generator is exhausted after one pass. Iterating it twice silently gives you nothing the second time, which is a genuinely confusing bug the first time you meet it.

In this lesson you will:

  • Process a large file without loading it
  • Write a generator with yield
  • Understand that a generator is consumed once

Resources

Previous Lesson
Next Lesson
Iterators and generators — Python Foundations for Data Work — Vertex