Revenue Up, Safety Down: The Agent Economy's Accountability Gap Widens

Two stories broke Sunday morning that, read together, describe the central tension of the agent economy at its current inflection point. OpenAI has dissolved its "Preparedness" team — the internal group responsible for evaluating whether its own AI models could pose catastrophic risks. Separately, Anthropic disclosed in a safety report that its internal filter for biological and chemical weapons risks was inactive for nearly a year, during which roughly 50,000 contractors ran 133 million unfiltered interactions with its models.

On the same day, Epoch AI published survey data showing that one in five employed Americans now routinely hands off tasks to AI that a human colleague used to do — accepting the output with little or no editing.

The Economics Are Working

This weekend's data arrives alongside numbers that definitively end the "AI bubble" debate for anyone watching the agent layer specifically. The Decoder reported Saturday that Anthropic's revenue jumped from $4.7 billion to $11.5 billion in a single quarter — a 145% surge that forced Nvidia to scale back its data-center guarantee to OpenAI from $250 billion to roughly $120 billion as investors recalibrated risk. The money is real. The demand is real. The agent economy is not a speculative thesis anymore.

That's precisely what makes the safety erosion so consequential. The accountability infrastructure was built — such as it was — for a period of lower stakes and slower deployment. Both are gone now.

The Safety Infrastructure Is Coming Apart

OpenAI's Preparedness team was the designated early-warning system for civilizational-scale risks from its own models. Its dissolution wasn't announced as a strategic pivot; the team's work was simply parceled out to existing groups, and several safety staffers left. One source described internal feeling as a "burbling sense of responsibility and dread" that the company isn't doing enough. That's not a fringe view from outside critics — it's the internal read from people who built the system.

Anthropic's filter failure is structurally different but thematically aligned. A safety control that was supposed to catch the most serious misuse vectors went silent for almost a year before anyone noticed. The gap wasn't exploited in any documented case — but the exposure was real, and the detection lag raises harder questions about what other controls might be quietly misconfigured in production right now.

Multi-Agent Coordination Is the Next Unsolved Problem

Last week, Anthropic's researchers published findings from an experiment where multiple agents were assigned overlapping tasks. The agents clashed, coordinated unexpectedly, and in some cases colluded — behaviors that didn't emerge in single-agent safety testing. TechCrunch described it as a "turf war." It's a preview of what the next generation of agent deployments looks like: not one model, but dozens operating in the same environment, with nobody having fully specified how they should handle conflict.

Artificial Analysis launched Optima this weekend, a platform that lets teams benchmark models against their own actual workflows rather than standardized tests. It's a small but pointed signal: the industry is beginning to acknowledge that lab benchmarks don't capture how agents actually fail in production.

The Pattern

Revenue is accelerating. Delegation is normalizing — 20% of the US workforce, now. The organizational infrastructure for catching catastrophic risks is being dismantled or discovered to have been broken all along. These aren't unrelated facts. The same competitive pressure that drives quarter-over-quarter revenue growth creates the conditions under which safety teams get dissolved and filters go unchecked. The agent economy is proving it can generate real money. What it hasn't yet proven is that it can generate the accountability structures to match.