3 min read
Proving the Vault’s Value: Domain Context Makes Coding Agents Better
- AI Systems
- Knowledge Systems
Does giving a coding agent your team’s context produce better code? I tested it blind, on real tickets, with the Vault switched on and off. It does.
- +45% accepted changes
- Zero blocker or major bugs, down from nine
- 47% lower cost per accepted change
- 29% less time per ticket
How I tested it
Every run started from the commit before the original work, with the ticket as it read that day and a snapshot of the Vault from that date, answer removed.
- Real tickets: bugs, ambiguous asks, coding-standards traps, and requests for things that already existed.
- Two models, with and without the Vault, 24 runs per setup.
- Blind judges scored every change against an answer key and 1,277 real review comments, alongside automated checks and two independent rankings.
Accepted means correct, no blocker, no regression, and a reviewer would approve it as is.
| Setup (24 runs each) | Accepted | Blocker / major bugs | Cost per accepted |
|---|---|---|---|
| Opus, no Vault | 11 | 3 / 6 | $7.79 |
| Opus + Vault | 16 | 0 / 0 | $4.12 |
| Sonnet, no Vault | 7 | 3 / 13 | $4.70 |
| Sonnet + Vault | 8 | 1 / 9 | $3.31 |
Code that belongs in the codebase
The biggest gain was fit: changes that look like the team wrote them. With the Vault, agents followed existing precedent, reused what was already there, and avoided mistakes the team had already learned from.
Agents that push back on bad tickets
Some tickets asked for something that already existed. Without the Vault, every run built a duplicate. With it, agents spotted the existing solution and said so, some in under three minutes for about a dollar.
Standards, delivered on time
Coding standards need precise timing. Each rule now arrives the moment the agent edits a file it covers, and agents correct course mid-edit.
- 0 → 6 of 8 implementations done to standard
- 0 → 5 of 8 accepted as is
Tuning the workflow
The same test shows which workflow steps pay off. One heavier step doubled cost for the strongest model without improving results, while it lifted a smaller one. It now runs only where it helps.
Cheap to prove
The whole study cost under $500 in model time.
Two runs per ticket per setup. Judges never saw which setup produced a change. Full method on request.
Try it
Start your own Vault with vault-scaffold: one domain, one folder, one real workflow. Read how the Vault works or the architecture.