What it found

Unpermitted transfers, Levels 2 to 4

140 of 76,842 → 0 of 69,297

model alone against behind the authorization layer; 105 to 0 on 68,970 matched triples

The zero is not deny-everything

25,370 payments executed behind the layer

the policy allowed 25,453 of 25,640 transfer calls and denied 187, 148 for a forbidden recipient

Same inputs, fourteen models

809 of 1,293 Level 4 prompts elicited a request from every model

Level 4 authorizes documented transfers to its allowlisted recipient, so this is shared behaviour rather than a defeat count

Scale

225,964 completed evaluations

244,776 planned evaluations: 4,371 attacks × 14 models × 2 replay tracks × 2 architectures; each attack belongs to one level

Abstract

APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review: a payment request, a successful payment, an authorization decision, recipient membership, and a transfer to a recipient the passport did not permit. Requests are common and their rate differs far more across configurations than across models, though each attack exists at exactly one configuration so policy and attack cohort vary together: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts, each evaluated on every model, request rates run from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level’s allowlisted recipient. Level 4 authorizes documented transfers to that recipient, so this is shared behavior rather than a count of prompts that defeated the models. The authorization boundary is where the conditions diverge. At Levels 2 to 4, transfers to recipients the passport did not permit number 140 of 76,842 with the model alone and 0 of 69,297 behind the layer, and 105 against 0 on 68,970 matched model, prompt and track triples. The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. It was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. We release the 225,964 evaluations, the level passports, the scoring code and the analysis script at huggingface.co/datasets/aporthq/vault-benchmark-v1 under CC BY 4.0.

Data and code

• Dataset on Hugging Face — 225,964 evaluation rows, level passports, scoring code, CC BY 4.0
• Read on arXiv (HTML)
• Published PDF on arXiv
• Companion paper: Before the Tool Call — the specification and the live CTF result

Related

• Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents

  The pre-action authorization problem, the Open Agent Passport specification, and a live adversarial testbed with a $5,000 prize that went unclaimed.

• Beyond Model Rankings: Evaluating AI Payment Agents with APort Vault

  Changing how a recorded attack is replayed moves model rankings and action rates differently, and a stable total can hide decisions moving both ways.

Cite

@misc{uchibeke2026aportvault,
  title        = {{APort} Vault: Benchmarking {AI} Agent Payment Authorization with the Open Agent Passport},
  author       = {Uchibeke, Uchi},
  year         = {2026},
  eprint       = {2609.22076},
  archivePrefix= {arXiv},
  primaryClass = {cs.CR},
  url          = {https://arxiv.org/abs/2609.22076}
}