Handling latency and cancellation

Stop paying for answers nobody will read

When a user navigates away or edits their question, the in-flight generation is worthless. An abort signal threaded from the client through your route to the provider stops the billing as well as the work.

Cancellation has to propagate all the way. Dropping the response while the upstream call continues generates the full answer and charges you for every token of it.

Set a timeout as well. A stalled connection produces a spinner that never resolves, which is worse for the user than a clear failure.