The right context, at the right moment, at the right cost
A production agent has a few seconds and a token budget per turn. What it's handed, and when, decides the answer as often as the model does. Four levers matter.
1 · Fast
Context at the speed of conversation
A knowledge-search tool was too slow for a voice agent to use. I made it fast enough for live conversation without changing a single answer.
Some context can be fetched before the agent needs it. The catch is that background warming competes with live turns: one burst of it could stall unrelated live conversations. I built priority lanes and a cross-pod in-flight limiter, so warming waits in a queue instead of on the live path.
−65%
latency at the median
−64%
latency for the slowest 10%
−59%
on the first call after startup
100%
identical answers
2 · Lean
Every token earns its place
Context bloat costs money and accuracy. Fixing it starts with knowing what the model actually receives, which is often not what anyone assumes.
99%+
Taking apart a runaway request
A chat agent ran up a huge token bill on one ordinary question, and the failed tool calls looked guilty. Per-call token logs and traces told a different story: the total was two model calls added together, and over 99% of the first call's context had arrived before any tool ran, inherited from conversation history. The fix belonged in the history, not the tools.
+0%
Measure before you cut
Forwarding full output schemas to agents looked like a big token cost. I A/B-tested it: input tokens didn't move, because the agent framework keeps schemas as metadata and never sends them to the model. The effort went somewhere that mattered.
3 · Right
Help the agent make the right call
An agent chooses tools from what the tools say about themselves. Most bad tool calls trace back to a tool that described itself badly.
Never send a question to the wrong tool
The wrong tool means a confident, wrong answer. Every tool states exactly what it returns, and routing is measurable: any change to a tool, a prompt, or the model can be scored on whether questions still land in the right place, before a customer finds out.
Forgive mistakes only where provable
Agents call tools with the wrong argument names, and those calls simply failed. I added repair that renames an argument only when the mapping is provable, and about 9% of failed tool calls now recover on their own. It never guesses and never drops a filter, so a recovered call can't return the wrong answer.
4 · Cheap
The right context is the cheaper context
Every token in context is paid for on every turn, at production traffic. The cheapest context is the piece the model never needed. The most expensive is the piece it was missing, because that becomes a retry, a wrong answer, or a hand-off to a person.
47% less
Cost per accepted change when coding agents got the right context. Same model, same tickets.