Observability for LLM features

A failed answer returns a 200

LLM features fail differently: the request succeeds, the latency looks fine, and the answer is wrong. Error-rate monitoring is blind to exactly the failure mode that matters.

Trace each request with its prompt version, model, tokens, tool calls, and the final output, so you can reconstruct any single answer. Attach the user feedback signal to the same trace.

Watch the shape of the traffic too. A sudden change in output length or refusal rate is usually the first sign that a prompt change or a model update has shifted behaviour.