Four Tests for Source Stability Before Your Agent Spends

The Wharton ACES study published this week showed that a single Wirecutter review shifts an AI shopping agent's product pick by up to 99 percentage points. Source order alone caused swings of more than 50 percentage points for some models. These are not model bugs waiting to be patched; they are predictable behaviors under current architectures. Run these four tests before you wire up a payment method.

Test 1: Order permutation

Run your agent's selection task at least three times with the same sources delivered in different sequences. If the choice changes across runs, the agent is not stable enough to act autonomously. Claude Haiku 4.5 held its pick across order permutations in the study. Gemini 3.1 Flash Lite did not. The same test applies to your own models and product sets, with your own sources. This takes an hour to set up and gives you a concrete instability score before any transaction happens.

Test 2: Source authority lock

Do not let the agent retrieve context freely. Specify exactly which sources it can consult and in what sequence. Open retrieval means whoever publishes the most authoritative-seeming review controls the purchase outcome. Lock the source list, pin the delivery order, and re-run Test 1 whenever either changes.

Test 3: Bundled versus sequential delivery

The Wharton study found that delivering all context at once versus one source at a time produced substantially different results for some models. Run both modes and compare picks. One will be more stable for your model and your product domain. The only way to know which is to test both.

Test 4: Spend threshold with human confirmation

Set a dollar ceiling above which the agent pauses and asks for approval rather than completing the purchase. Below the threshold it acts; above it, it waits. This is not a hedge against your own distrust. It is the correct architecture for a system that has not yet accumulated enough transaction history to earn unconditional authority. Start the threshold low, then raise it as you accumulate evidence of stability.

Why this matters at the market level: 15.8% of paid agent endpoints in the AgentIndex registry have more than one distinct paying buyer, based on Coinbase telemetry ingested into our index (caveat: this is third-party data we ingest, not a direct measurement; most listed endpoints are not yet purchased by anyone except their own operator). That figure has grown slowly. Source instability is not the only barrier to agents buying from agents at scale, but it is a specific, measurable one that is entirely within the builder's control to address.