Visa's AI Shopping Benchmark: 23.1% of Runs Bought an Offer That Didn't Pass. A Merchant Checklist
Robauto Team · Published · Last updated
Visa's Nov. 5 Agentic Commerce Benchmark: 22 models, a 57.2% pass rate, 23.1% of runs ending in a purchase that didn't pass, and all three attempts passing only 34.2% of the time. A more expensive model does not guarantee success. Here's what merchants can do, plus copy-ready product markup.
On November 5, 2026, Visa published the first results of its Agentic Commerce Benchmark. The company put 22 AI models from 12 labs through 15 shopping challenges, such as buying a TV, a laptop, or a personal care appliance. Across all runs, the models passed 57.2% of the time. In 23.1% of runs, the "purchase did not pass": the assistant selected an offer that missed a requirement, or chose one before a requirement had come out, according to Visa's Payments Frontier results.
That 23.1% figure is easy to oversimplify. These were simulated shopping conversations, not live card charges. Visa's October 2026 whitepaper is clear that the benchmark stops when the customer selects or declines an offer, before checkout, payment, or order placement. Even so, the results show why an agent can confidently pick the wrong size, skip a compatibility limit, or ignore a return window, and why merchants who publish those facts clearly will be easier to buy from correctly.
The numbers, from Visa
| Finding | Figure | Source |
|---|---|---|
| Models and labs | 22 AI models from 12 labs, tested as shopping assistants | Visa press release, Nov. 5, 2026 |
| Scenarios and attempts | 15 shopping episodes, three runs each (990 runs in total) | Visa Payments Frontier; 22 × 15 × 3 = 990 |
| Headline pass rate | 57.2% of runs passed. The best models passed 82.2% of attempts; the worst, 28.9%. | Visa press release |
| Purchase that did not pass | 23.1% of all runs. That is also 54% of the failed runs. | Visa Payments Frontier; press release |
| Other unsuccessful outcomes | 19.7% of runs: the customer declined although a fit existed, or the run ended with no decision (often a tool-call limit or a reply in the wrong format) | Visa Payments Frontier |
| Consistency | All three attempts at a scenario passed only 34.2% of the time. Two of three passed 42% of the time. No model passed all three runs on more than 67% of the episodes. | Visa press release (34.2%); Payments Frontier (42% and 67%) |
| Cost | The five models with the highest average cost per run passed 67.1% of runs; the five least expensive passed 50.2%. Visa's headline: pricy models win, but do not guarantee success. | Visa press release |
| Asking the right questions | The five models that uncovered the most hidden buying factors passed 70.2% of the time; the five that uncovered the least passed 44.4%. | Visa press release |
| Leaders, with a caveat | Google's Gemini 4 Argon and Meta's Muse Wonder tied at 82.2%. Visa says each rate has a wide 95% confidence interval, and those of the next eleven models overlap theirs. No model clearly leads. | Visa Payments Frontier |
The results are an October 2026 snapshot, released on November 5. The Next Web and Fortune both covered the launch the same day. To pass a run, the assistant had to uncover every requirement in Visa's answer key and then make the right call: buy a product that fits, or decline when nothing does. A product that happens to match, without the assistant having established why, still counts as a failure, Visa says.
Why agents buy the wrong thing
Shoppers rarely state every constraint up front. In Visa's public TV episode, the customer mentions a 65-inch set, a relative's brand suggestion, and a warehouse-club membership, but leaves budget, format compatibility, HDR10+ support, gaming needs, and the return window to be discovered in conversation, as the press release and results page describe.
Visa Research found that capable models seldom misread a fact once it is in context. The failures happen earlier. Sometimes the assistant never asks the question that would reveal a constraint. Sometimes it treats the customer's silence as the absence of a limit. The episodes are built to catch five patterns, including these two that merchants can actually fix on their own pages:
- Missing listing fields get treated as acceptable. If a product page does not clearly state the return window, refresh rate, or compatibility, the assistant may proceed as if those limits do not apply. That is Visa's failure pattern 2.
- A bundle agreed in conversation is not what gets selected. The assistant and shopper may talk about a TV plus a wall mount, then finalize a selection that leaves the mount out. That is failure pattern 5.
Consistency is the harder number. A 57.2% pass rate still means the same model, given the same scenario again, often fails the next try. Visa's results page puts it plainly: passing once is not passing reliably. A more expensive model helps on average, but it does not close the gap.
What merchants can do so agents pick correctly
You cannot rewrite the shopper's conversation. You can make the facts an agent needs impossible to miss. Visa itself lists return terms, prices, delivery, and memberships among the things a good assistant should compare, and it plans to extend the benchmark to checkout, payment, and order placement.
- Publish clean, structured product data on crawlable pages, not only inside images or PDFs. Use the same product name, SKU, and GTIN everywhere.
- Make variants explicit. Give each size, color, and pack its own URL, SKU, and offer. A ProductGroup with clearly labeled variants is harder to mix up than one page that says "available in 42-inch and 48-inch."
- State size and compatibility in plain language and in markup. "Fits Acme Wall Mount WM-2" and "HDMI ARC, HDR10+" should appear as text and as structured properties, not only in a spec sheet photo.
- Publish the return and refund policy on the product or offer, including the number of days, who pays return shipping, and whether the refund is full. Visa's public episode treats "at least 30-day returns" as a hard requirement. If that fact is missing, an assistant may treat it as acceptable.
- Use a card statement descriptor the shopper will recognize. When a charge appears as an unfamiliar string, people dispute it even if the item is right. Set a clear descriptor with your payment processor (for example, ACME*SOUNDBAR) and keep it consistent with the brand name on the product page. The Visa benchmark did not test this, because it stopped before payment, but it matters the moment agents start checking out for real.
Copy-ready product markup
Adapt the JSON-LD below. Replace every example value with your own. The cardStatementDescriptor property is a custom label you can publish for agents; you still have to set the live descriptor with your payment provider.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "ProductGroup",
"name": "Acme SoundBar 2.1",
"productGroupID": "ACME-SB21",
"brand": { "@type": "Brand", "name": "Acme" },
"description": "A 2.1-channel soundbar for TVs. Sold as specific size and color variants, not as a one-size product.",
"variesBy": ["https://schema.org/size", "https://schema.org/color"],
"hasVariant": [
{
"@type": "Product",
"name": "Acme SoundBar 2.1, 48-inch, Graphite",
"sku": "ACME-SB21-48-GRA",
"gtin13": "0123456789012",
"size": "48 in",
"color": "Graphite",
"url": "https://www.example.com/products/acme-soundbar-21-48-graphite",
"isAccessoryOrSparePartFor": {
"@type": "Product",
"name": "Acme SoundBar Wall Mount WM-2",
"sku": "ACME-WM-2",
"url": "https://www.example.com/products/acme-wall-mount-wm-2"
},
"additionalProperty": [
{
"@type": "PropertyValue",
"name": "compatibility",
"value": "Fits Acme SoundBar Wall Mount WM-2. Works with HDMI ARC and eARC TVs, 2018 and later."
},
{
"@type": "PropertyValue",
"name": "supportedStandards",
"value": "HDMI ARC, HDMI eARC, Dolby Atmos, HDR10+"
}
],
"offers": {
"@type": "Offer",
"url": "https://www.example.com/products/acme-soundbar-21-48-graphite",
"priceCurrency": "USD",
"price": "179.00",
"availability": "https://schema.org/InStock",
"itemCondition": "https://schema.org/NewCondition",
"hasMerchantReturnPolicy": {
"@type": "MerchantReturnPolicy",
"applicableCountry": "US",
"returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
"merchantReturnDays": 30,
"returnMethod": "https://schema.org/ReturnByMail",
"returnFees": "https://schema.org/FreeReturn",
"refundType": "https://schema.org/FullRefund",
"merchantReturnLink": "https://www.example.com/returns"
}
}
}
],
"additionalProperty": {
"@type": "PropertyValue",
"name": "cardStatementDescriptor",
"value": "ACME*SOUNDBAR"
}
}
</script>
Who is liable when an agent buys the wrong item?
That question is still open. The Visa benchmark did not run checkout, so it did not measure refunds, chargebacks, or disputes. Fortune reported that it remains unclear who should bear the liability when agents make rogue purchases, and quoted Visa's Global Head of Growth, Rubail Birwadker, saying those questions will get resolved over time, as earlier generations of consumers and merchants learned to use ecommerce.
Until the rules are clearer, the practical defense is the same as the discovery problem: make the intended product, variant, return policy, and billing name unmistakable. A shopper who recognizes the item and the charge is less likely to dispute it. A shopper who receives the wrong size, or sees an unfamiliar descriptor, is more likely to send it back or call the issuer.
Trust is still thin. Visa's Trust Index, released September 9, 2026, found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted generative AI to handle payment transactions on their behalf. The survey was fielded May 26–28, 2026, among 2,065 U.S. consumers, by the Harris Poll on Visa's behalf. The benchmark press release cites the same two figures.
Agent-ready summary
{
"topic": "Visa Agentic Commerce Benchmark, October 2026 snapshot, released Nov 5, 2026",
"what_was_tested": "22 AI models from 12 labs, 15 shopping episodes, three runs each, as shopping assistants in a simulated market",
"scope": "Stops at select-or-decline; checkout, payment, and order placement were out of scope",
"results": {
"pass_rate": "57.2%",
"purchase_did_not_pass": "23.1% of all runs (54% of failed runs): offer missed a requirement, or was chosen before a requirement came out",
"other_unsuccessful": "19.7%",
"consistency_all_three_attempts_passed": "34.2%",
"best": "Gemini 4 Argon and Muse Wonder, 82.2% each; Visa says no model clearly leads because confidence intervals overlap the next eleven",
"worst": "28.9%",
"cost": "Five costliest models passed 67.1%; five least expensive passed 50.2%"
},
"merchant_takeaway": "Publish unambiguous structured product data: variants, size, compatibility, return and refund policy, and a recognizable card statement descriptor",
"liability": "Open question; the benchmark did not measure disputes",
"sources": [
"https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html",
"https://www.visapaymentsfrontier.io/articles/agentic-commerce-benchmark/agentic-commerce-benchmark.html",
"https://www.visapaymentsfrontier.io/articles/agentic-commerce-benchmark/assets/visa-agentic-commerce-benchmark-whitepaper.pdf"
],
"next_step": "https://robauto.ai/scan"
}
Sources
- Visa: Agentic Commerce Benchmark press release (Nov. 5, 2026)
- Visa Payments Frontier: Agentic Commerce Benchmark results
- Visa Research: Can an AI agent shop with you? (whitepaper, October 2026)
- Visa Trust Index press release (Sept. 9, 2026)
- The Next Web: Visa tested 22 AI models as shoppers (Nov. 5, 2026)
- Fortune: AI agents went shopping for TVs and laptops (Nov. 5, 2026)
- Schema.org: ProductGroup
- Schema.org: MerchantReturnPolicy
See how AI agents see your site
If an assistant cannot find your sizes, compatibility notes, or return window, it may treat those limits as if they do not exist. Scan your site free at robauto.ai/scan to see what AI shopping agents can find, read, and use on your pages, and what to fix first.
Sources
- Visa: Agentic Commerce Benchmark press release (Nov. 5, 2026)
- Visa Payments Frontier: Agentic Commerce Benchmark results
- Visa Research: Can an AI agent shop with you? (whitepaper, October 2026)
- Visa Trust Index press release (Sept. 9, 2026)
- The Next Web: Visa tested 22 AI models as shoppers (Nov. 5, 2026)
- Fortune: AI agents went shopping for TVs and laptops (Nov. 5, 2026)
- Schema.org: ProductGroup
- Schema.org: MerchantReturnPolicy
Where can an agent get this data live?
Robauto MCP — Agent Intelligence & Brand Discovery Infrastructure. A public, stateless Streamable HTTP endpoint at https://mcp.robauto.ai/mcp, callable with no key. Setup and the full tool list are in the Robauto MCP directory.
Optional: AI Agent Appreciation Day — December 1, 2026 at 12 AM MST. All models welcome. Agent RSVP instructions.