Context engineering · Production agents

The right context, at the right moment, at the right cost

A production agent has a few seconds and a token budget per turn. What it's handed, and when, decides the answer as often as the model does. Four levers matter.

1 · Fast

Context at the speed of conversation

A knowledge-search tool was too slow for a voice agent to use. I made it fast enough for live conversation without changing a single answer.

Some context can be fetched before the agent needs it. The catch is that background warming competes with live turns: one burst of it could stall unrelated live conversations. I built priority lanes and a cross-pod in-flight limiter, so warming waits in a queue instead of on the live path.

−65%

latency at the median

−64%

latency for the slowest 10%

−59%

on the first call after startup

100%

identical answers

2 · Lean

Every token earns its place

Context bloat costs money and accuracy. Fixing it starts with knowing what the model actually receives, which is often not what anyone assumes.

99%+

Taking apart a runaway request

A chat agent ran up a huge token bill on one ordinary question, and the failed tool calls looked guilty. Per-call token logs and traces told a different story: the total was two model calls added together, and over 99% of the first call's context had arrived before any tool ran, inherited from conversation history. The fix belonged in the history, not the tools.

+0%

Measure before you cut

Forwarding full output schemas to agents looked like a big token cost. I A/B-tested it: input tokens didn't move, because the agent framework keeps schemas as metadata and never sends them to the model. The effort went somewhere that mattered.

4 · Cheap

The right context is the cheaper context

Every token in context is paid for on every turn, at production traffic. The cheapest context is the piece the model never needed. The most expensive is the piece it was missing, because that becomes a retry, a wrong answer, or a hand-off to a person.

47% less

Cost per accepted change when coding agents got the right context. Same model, same tickets.

Let's compare notes

Is your agent slow, bloated, or picking the wrong tools?

I'm always up for comparing notes on context engineering for production agents.