Your Harness Sets the Bill, Not Your Model

Two researchers working through the MATS program asked coding agents a question that sounds trivial and is not: do you know how long you have been working? They tested Anthropic's Claude Code and OpenAI's Codex across 200 tasks from ProgramBench plus 18 benchmarks of their own, asking each agent to estimate duration before starting and to report elapsed time afterwards. The agents were wrong in both directions and wrong by a lot. Both mostly guessed around 90 minutes no matter how hard the task was. Claude Code overshot actual runtime by roughly three times on average. Codex overshot by six to ten times. The error was worst on short tasks, which is to say on almost everything an agent does in production.

The finding that should reorganise how you budget is the next one. The same underlying model took 2.5 times more steps in Claude Code than in Codex, because Claude Code keeps going until it decides the job is finished, a median of about 90 minutes, while Codex stops after roughly half an hour almost regardless of the task. Same model, same task, wildly different amount of work done. The wrapper decided, not the intelligence inside it.

That matters commercially because steps are the unit that gets billed. Across the paid endpoints our index tracks, the median charge is $0.01 per call. That figure carries its own warning: 5 implausible prices were excluded (the largest records 10bn USDC) and 98 free endpoints were held out of the percentiles. But the shape holds. Agent spending is not priced by how smart the reasoning was, it is priced by how many times the agent reached for something. An agent that cannot tell whether it has taken twelve steps or two hundred is an agent with no felt sense of its own bill, and it is sitting inside a harness that quietly sets that number for it.

The study's second result closes the trap. Both systems rated their own output about 20 percentage points above what they actually scored. In one run both judged themselves near 70 percent successful when the real numbers were 7 and 14.5 percent. So the component that cannot measure its spend also cannot be trusted to say whether the spend bought anything.

Three things to change this week.

Feed time in rather than asking for it. Every tool result your harness returns should carry wall clock time, elapsed seconds since the run began, and the running step count. This is a string concatenation, not a research project, and it converts an inference problem into a lookup.

Budget in calls, not in minutes or dollars. Put a hard counter in the harness that refuses the next paid call past a ceiling. The agent's own estimate is a guess, and the study says it is a bad one, so it cannot be the control.

Never let the agent grade its own purchase. Success has to come from the endpoint's response, a receipt, a status code, a schema that either validates or does not. A model summarising that it went well is the least reliable signal you have access to.

If your workload spends money, choose the harness before you choose the model. One of those decisions sets your capability. The other sets your invoice.