Cost, caching, and token budgets

Cost is a design constraint

Token spend scales with usage, so a feature that is cheap in testing can be expensive at launch. Measure cost per request early, while the design is still easy to change.

Prompt caching pays for itself whenever a long, stable prefix is reused — system instructions, a schema, a document. The variable part goes at the end so the cached prefix stays identical.

Not every step needs the largest model. Routing classification and extraction to a smaller one and reserving the large model for synthesis often cuts cost several-fold with no measurable quality loss.