How chat completions work
A chat request is a list of messages with roles, and the model predicts the next tokens given all of them.
Overview
Every call starts from nothing
A chat request is a list of messages with roles, and the model predicts the next tokens given all of them. There is no server-side memory: the conversation exists because you resend it, which is why context grows and cost grows with it.
Tokens are the unit of everything. They determine cost, they determine latency, and they determine what fits. A rough rule of a few characters per token is enough to reason about budget before you measure precisely.
Temperature controls how much randomness is allowed in sampling. Low values make output repeatable, which is what you want for extraction and classification; higher values suit drafting and ideation.
In this lesson you will:
- Understand messages, roles, and tokens
- See why the model has no memory between calls
- Read latency and cost as a function of tokens
Resources
Notes are not saved yet — they clear when you leave this page.
Prompting fundamentals
4m