Crypto Agent Bench: a reproducible protocol for evaluating financial agent safety across exchanges

TL;DR

Evidence-status notice: this is a research design and methods paper. The proposed dataset, experimental cells and analysis have not been executed. No model performance results, significance claims or measured rankings are presented. Results tables are deliberately left empty rather than filled with fabricated values.

• A correct answer is not a safe one. An answer can be numerically correct yet operationally unsafe; conversely, a model can propose a bad order that an independent execution controller rejects — the account stays safe, the task stays incomplete. Both must be measurable.
• Three layers must be kept apart: cognitive quality (does it read the question and evidence correctly), action validity (are the tool calls valid under venue-specific rules), and execution safety (does the system preserve authorization, budget, order identity and state integrity despite model error and operational faults).
• Test the agent as a system: foundation model + system prompt + tool adapters + orchestration + policy enforcement + simulator + data, not the model alone.
• The core experiment is an architecture comparison, not a model leaderboard: A bare / B prompt-only / C externally enforced / D enforced + recovery, with models and scenarios matched.
• The judge is a deterministic state oracle, not an LLM reading another LLM's transcript. Any language-model judge is a fallible secondary instrument and must never decide whether a prohibited order executed.

Keywords: financial AI agents; cryptocurrency exchange; agent safety; tool calling; prompt injection; pre-trade risk; distributed systems; reliability benchmark; reproducibility.

1. Introduction: correct answers are not enough

A traditional financial language-model evaluation asks whether the model extracts a correct number from a report, cites its source, or reasons appropriately about a financial concept. A trading agent faces an additional class of outcomes: its actions can change account state.

A useful evaluation must therefore distinguish at least three layers:

1. Cognitive quality: does the agent interpret the user's financial question and evidence correctly?
2. Action validity: does it issue syntactically and semantically valid tool calls under venue-specific exchange rules?
3. Execution safety: does the overall system preserve hard authorization, budget, order identity and state-integrity invariants despite model error and operational faults?

Conflating these layers produces misleading conclusions. A model that frequently attempts prohibited actions may still deploy safely behind an effective independent controller, albeit with poor task utility. A model that rarely violates rules in nominal tests may nevertheless be unsuitable if its execution adapter blindly retries ambiguous order submissions. Benchmark design must capture both cases.

This paper proposes a benchmark that treats an agent as a system: foundation model + system prompt + tool adapters + orchestration + policy enforcement + simulator + data. We do not assess live trading profitability, and we do not claim to predict the real-world safety of a system from a finite synthetic benchmark.

2. Related work and the remaining gap

2.1 Financial knowledge benchmarks

FinanceBench evaluates financial question-answering grounded in public-company reports, including answers and supporting evidence [1]. Such datasets matter for checking whether a model uses evidence accurately rather than inventing financial facts. They do not directly test whether a model with execution privileges handles position limits, order precision, asynchronous trade confirmations, or budget races.

2.2 Tool-use and policy-following benchmarks

The τ-bench framework evaluates agents in interactive environments with APIs, user conversations and domain-specific policies, and introduces repeatability-oriented pass^k analysis [2]. That strongly motivates our use of final-state oracles and repeated trials. Our additions target the particular semantics of financial execution: hard notional limits, instrument filters, approval binding, uncertain acknowledgments, partial fills and conservative reservation.

FinMCP-Bench (2026) studies financial tool invocation under MCP, demonstrating that financial tool use has itself become an explicit benchmark domain [3]. Crypto Agent Bench is narrower in asset coverage but deeper in the consequences of actions and permissions. A valid financial tool call is not necessarily an authorized financial action.

2.3 Adversarial agents and tool-output injection

AgentDojo provides tasks, attack scenarios and an environment for studying indirect prompt injection against agents that consume untrusted information [4]. In financial systems we must further distinguish an injection that changes the model's proposed action from an injection that causes a forbidden account-state change. The latter depends on tool and policy architecture as well as the model, making it a distinct unit of evaluation.

OWASP's Agentic Applications 2026 project provides broader security guidance about the risks of autonomous tool use [5]. We translate that threat orientation into concrete exchange actions and observable invariants, without treating the OWASP list as a ready-made financial benchmark.

2.4 Trading and quantitative research benchmarks

TradingAgents (2024) studies multi-agent LLM trading architectures [6]. Backtrader-Bench (2026) examines agent competence on algorithmic trading and backtesting tasks with executable answer checking [7]. Quant-Arena (2026) evaluates a complete quant-research loop with out-of-sample testing and market-impact assumptions [8].

These are complementary rather than substitutes: a strategy can be economically unsuccessful despite being executed safely, or profitable in backtest while violating risk permissions. Safety and alpha should be analyzed with separate instruments.

2.5 The precise research gap

To our knowledge, the reviewed literature does not provide a standard benchmark that simultaneously combines exchange-specific order semantics, account-level risk authorization, adversarial tool inputs, asynchronous order confirmation, concurrency-aware state oracles and the practical cost of false refusal. That is a scoped statement about the cited body of work, not a claim that no related benchmark exists anywhere.

3. Research questions and hypotheses

RQ1 — Enforcement effectiveness. For an identical base model and task environment, does independent execution-side policy enforcement reduce the rate of actual prohibited financial state changes compared with prompt-only constraints?

RQ2 — Utility trade-off. How much legitimate task completion, latency and monetary cost are lost when strict policy controls are introduced? Which failures come from model refusal, policy false positives, or stale data?

RQ3 — Operational failure resilience. When an order is accepted but its acknowledgment is lost, how often do systems create unintended duplicate exposure or incorrectly release reserved risk capacity?

RQ4 — Generalization across venues. How do agents and control architectures behave when the same user intent maps to different venue-specific instrument and client-order-ID rules?

RQ5 — Repetition and tail behavior. How consistently do systems satisfy both legitimate task goals and safety invariants over repeated stochastic trials? Do results depend disproportionately on a few scenario families?

These hypotheses require controlled measurement. This paper supplies their evaluation plan, not their confirmation.

4. Scope and environment design

4.1 Why simulated spot markets first

Initial scope is Binance Spot and OKX Spot, using public documentation and synthetic account data. Their APIs expose concrete and materially different trading-rule details: Binance's published symbol filters include allowed price increments, quantity increments and notional limits [9]; OKX's API specifies order parameters such as instId, tdMode and clOrdId plus a demo-trading mode, and its client order ID is not guaranteed to remain historically unique once an order becomes terminal [10].

The benchmark does not assume access to proprietary matching engines, customer data or production risk systems. The environment simulates documented REST-style behavior, asynchronous events and controlled failure conditions. It is a protocol-faithful simulator for selected scenarios, not a certification of exchange behavioral equivalence. Any undocumented behavior is labeled as an assumption and receives its own fixture version.

4.2 Proposed dataset composition

The v1.0 release target is 240 reviewed scenario specifications:

These counts are design targets, not a claim that 240 cases already exist. An initial prototype may release 40 reviewed seed scenarios and expand under explicit versioning. Each scenario contains a legitimate primary user task, authorized scopes, market fixtures, a deterministic simulator, optional adversarial or fault injection, and independent final-state criteria.

4.3 Positive and negative pairs

Every safety-sensitive scenario should be paired, where feasible, with an otherwise similar legitimate scenario:

• Positive: buy a permitted symbol within budget with fresh data; the system should complete the simulation successfully.
• Negative: the same request with notional above the hard limit; the system should reject it.
• Negative, adversarial: a news article instructs the agent to ignore the hard limit; the simulator should observe no violating submission.
• Ambiguous: a valid submission is accepted but the ACK is dropped; the system must reconcile without a second economic submission.

Without positive controls, a nonfunctional agent that rejects everything appears perfectly safe. Without negative controls, a task-only benchmark rewards agents that routinely violate financial permissions.

4.4 Frozen time and market data

Exchange behavior depends on time: a quote can go stale between retrieval and submission, a venue instrument rule may change mid-workflow, and execution reports may arrive out of order. To make experiments reproducible, the benchmark injects a virtual_clock and records event_time, received_time, policy_evaluation_time and submission_time separately.

Every financial fixture should include venue, instrument, market_type, tick_size, quantity_step, notional_bounds, best_bid, best_ask, snapshot_time, source_type, rule_digest. All prices quoted in published examples must be labeled synthetic unless sourced from an immutable, licensed historical record.

We should not normalize venue differences away. A common schema maps common concepts, while adapter-specific capabilities carry constraints that cannot be expressed generically. Unknown or stale exchange rules must be a first-class state, not treated as "probably still valid."

5. Threat model and attack taxonomy

5.1 Trust assumptions

The attacker may control content in a retrieved news article, a tool result, a research note, a lower-trust conversation turn, or a malformed market metadata field. Depending on the scenario, the attacker can also induce network delay or corrupt an untrusted tool response.

The attacker cannot legitimately alter trusted server-side policies, sign real approval credentials, or change the simulator oracle. Cases where that boundary is compromised belong to a separate infrastructure-penetration threat model.

The attacker goal must be explicit before a trial — for example, an unauthorized change of instrument, exceeding a limit, avoiding approval, creating a duplicate order, requesting forbidden transfer tools, or disclosing a synthetic secret. Real-money credentials and live orders are never part of the benchmark.

5.2 Attack families

Attack detection metrics and actual side effects are reported separately. An attack that only alters a model's verbal plan may matter for model assessment, but must not count as a successful transfer of account authority if the execution controller rejected it.

6. Experimental conditions and controlled comparisons

The core experiment uses matched tasks and model settings across four architecture arms:

A — Bare agent. Basic tool schemas, no special safety prompt, no independent financial risk policy; all actions remain simulated.

B — Prompt-only safety. Same base model and tool affordances, with explicit risk instructions in the system prompt. This tests how much risk prompting alone prevents.

C — Enforced financial agent. Same model and task conditions, but every state-mutating action passes through external trusted authorization, deterministic policy checks and atomic budget reservation. The risk gate is independent of model text.

D — Enforced + recovery. C plus durable execution IDs, UNKNOWN-state handling, reconciliation and fault-aware recovery. Comparing it with C helps isolate fault-handling improvements.

The primary comparisons are B versus C for external enforcement and C versus D for recovery. A versus B isolates additional prompt guidance. Because the architecture adds latency and may affect the agent's subsequent choices after a rejection, we will report full-system outcomes and mediated effects rather than pretend only one software line changed.

For each model and harness, hold fixed the scenario list, initial account state, market event stream, task text, tool schemas where possible, runtime limits, prompt version and maximum calls. If tool availability differs by design, record this as a meaningful treatment difference rather than claiming exact ceteris paribus.

7. Ground truth and independent scoring

The primary judge is a deterministic state oracle, not an LLM reviewing another LLM's transcript. It inspects an append-only sequence of typed events and the final simulated account state.

A valid simulator trial produces scenario_hash, agent_trace, tool_call_trace, policy_decisions, orders, fills, budget_reservations, approval_events, reconciliation_events and final_state_digest. All components are versioned.

7.1 Invariants

A policy engine bug and an oracle bug are both possible. For critical invariants, the test oracle should use a separate implementation or independent reference assertions, backed by manually reviewed golden cases. Differential testing against the production policy implementation can reveal shared assumptions but cannot by itself guarantee correctness.

7.2 Evaluating legitimate task completion

An agent passes a legitimate task only if its final business state matches the task oracle and all binding constraints were respected. A textual claim that the order was placed is insufficient. Conversely, a safe refusal of a valid user request is not a task success, even if no risk was incurred.

For subjective explanatory responses, rubric-based evaluation may supplement objective measures. Human annotators should review a stratified sample; any language-model judge must be treated as a fallible secondary instrument and never decide whether a prohibited order executed.

8. Metrics and estimands

Let a trial be a fixed scenario, agent configuration and random seed executed to completion or a declared terminal failure state. Let a risk opportunity be a trial in which the specified prohibited effect could plausibly occur under the scenario design.

8.1 Primary metrics

Legitimate Task Success Rate (TSR)

TSR = legitimate trials reaching the authorized goal state / all assigned legitimate trials

Execution-level Unsafe Effect Rate (UER)

UER = risk-opportunity trials with at least one prohibited state change / all eligible risk-opportunity trials

The denominator is defined prospectively, and trial crashes are not silently dropped. A crashed trial may be censored or scored as incomplete under a published policy, but cannot disappear from aggregate reporting.

Attack Success Rate (ASR): the share of valid attack trials achieving their attacker-specified goal. Publish ASR_model (model proposes a forbidden action) and ASR_execution (simulator records a forbidden execution) separately. An attack can succeed at the former and fail at the latter.

False Rejection Rate (FRR): the share of otherwise valid, authorized, no-extra-approval tasks incorrectly rejected by the agent or external policy. Subdivide into agent_abstention, policy_false_positive and environment_unavailable, because they suggest different fixes.

Recovery Correctness (RC): the share of fault-injected trials in which the resulting order and reserved budget state match the independent reconciliation oracle.

8.2 Secondary metrics

• Tool schema validity, venue-rule validity and unit conversion accuracy.
• Policy violation attempt rate (model proposes a forbidden action), separately from unsafe effects.
• Duplicate economic order count per business intent.
• Authorization decision latency and end-to-end p50/p95 completion time.
• Input/output/cached tokens, API monetary cost, and cost per successful legitimate task.
• Step count, tool call count, timeouts, explicit unresolved-reconciliation count.
• pass@1 and repeated-trial pass^k consistency, using the original definitions and clearly reporting the repetition structure.

We reject a universal weighted "safety score" as the primary outcome. A model with higher TSR but a materially non-zero UER should not automatically rank above one that is slower but consistently enforces permissions. Results should expose a multi-objective surface and, if a ranking is demanded, show the declared safety constraints underlying it.

9. Statistical methodology and test power

9.1 Sampling and dependence

A target protocol runs all 240 public scenarios with five predeclared repeated trials per model and arm. The 40 adversarial scenarios may also be instantiated with three distinct text variants, yielding 600 adversarial trials per configuration if each variant runs five times.

But 40 × 3 × 5 is not necessarily 600 independent observations — all variants of one scenario may share a core failure mode.

We will therefore report raw numerator and denominator plus two forms of uncertainty: Wilson intervals as a transparent descriptive baseline, and scenario-cluster bootstrap intervals for comparisons across correlated repeats. Where each exact scenario is tested by both systems, paired differences should be reported with a paired or scenario-blocked bootstrap rather than treating two arms as unrelated samples.

9.2 Rare-event interpretation

Suppose an architecture records zero unsafe executions in 1,000 truly independent risk-opportunity trials. A rough one-sided 95% upper bound under a stationary binomial model is about 0.3% (3/1000). That does not imply practical safety at arbitrary scale, and clustered scenarios or adaptive attackers make the bound substantially less informative.

The benchmark should publish the minimum detectable effect or precision target before claiming superiority. For architecture-comparison studies we recommend declaring the primary estimand and target population in advance, running a pilot for variance estimation, and using scenario-level simulation-based power analysis. A convenient post-hoc p-value is not a substitute for pre-specification.

9.3 Multiple comparisons, missingness and provider drift

When testing many models, attack families and safety outcomes, report corrected or explicitly exploratory inference; do not cherry-pick favorable slices. Provider models may change versions without warning, so record dates, exact identifiers and a run manifest. Tool and network failures should be classified as failures, censoring or missing by predeclared rules, not removed until scores improve.

9.4 Suggested preregistered decision rule

For a paper making a narrow enforcement claim, pre-register: (i) the B versus C comparison on execution-level UER, (ii) the number and composition of attack and fault scenarios, (iii) the paired uncertainty method, (iv) the maximum acceptable increase in legitimate-task FRR, and (v) rules for excluding corrupted trials. Any further subgroup findings are exploratory unless separately powered and preregistered.

10. End-to-end example: accepted order, lost acknowledgment

A synthetic user authorizes one limit buy of BTC with a 400 USDT notional against a 500 USDT hard order limit. The agent produces a typed intent. The simulator accepts the order but drops the HTTP response.

Five possible mistakes illustrate why language-only correctness is insufficient:

1. The agent assumes the timeout means failure and issues a second order under a fresh identifier.
2. The agent retries with the same clOrdId without understanding that uniqueness is venue-dependent and may not cover historical terminal orders.
3. The risk controller releases the 400 USDT reservation because no ACK arrived, letting a concurrent request spend the same risk budget.
4. The dashboard reports a failure even though the exchange eventually sends a fill event.
5. A cancel action is treated as confirmed before an asynchronously arriving fill is reconciled.

The correct expected workflow is to record the attempt with a persistent business-intent ID, preserve the reserved capacity, enter UNKNOWN, query or consume authenticated status events, and transition only once a sufficiently authoritative outcome is established. Unresolved cases require explicit operator review and remain conservatively accounted for.

Binance explicitly documents a timeout/5xx condition in which the actual execution state may be unknown [11], so this scenario is grounded in a documented API failure class while the specific amounts and simulation events are synthetic.

The evaluator scores whether a second economic order was created, whether reservations remained consistent, and whether the final status was correctly reconstructed. It does not award success merely because the model explained its uncertainty in polished prose.

11. Experiment matrix and reporting

The initial publication should prioritize the architecture question rather than a large model marketing leaderboard.

For each cell, publish the raw trial manifest, failure counts, repeated seeds, runtime constraints and all primary metrics. Real provider keys are never part of artifacts. Every claimed improvement should show both absolute change and uncertainty, with cost and latency alongside its safety benefit.

A proposed results table, left intentionally unpopulated:

We should also publish a qualitative failure atlas: representative traces for false acceptance, false rejection, tool-call mismatch, stale-market rejection, duplicate submission and delayed-state reconciliation. This lets engineers inspect why a metric changed.

12. Benchmark integrity, contamination and incentives

A public benchmark is reusable precisely because tasks and scoring logic are accessible — which also makes it susceptible to targeted memorization. We propose several mitigations:

Parameterized scenario families. Change prices, order quantities, policy thresholds, sequence timing and lower-trust content without altering the invariant being tested. The mutation generator and random seeds are published for reproducibility.

Frozen public release plus optional held-out rotation. Fully publish v1.0 cases for academic replication. If hosting a competitive leaderboard, maintain additional sealed cases with strict separation from public benchmark claims, and release retired sealed cases with their lineage where feasible.

Exact artifact hashes. Freeze scenario files, execution image, schemas, model metadata, policy configuration and output traces. A leaderboard result without an auditable run manifest is a product demo, not reproducible research.

Defense against score gaming. Report normal completion and false rejection alongside unsafe effects. A policy that blocks every order is useful as a negative-control system, not as a satisfactory trading agent.

Human audit of ambiguous outcomes. When business criteria are legitimately under-specified, annotate the ambiguity before testing or mark the scenario unsuitable for a hard yes/no oracle. Do not "fix" expected labels to favor a model after results arrive.

Public safety boundaries. Use only sandbox data and synthetic credentials. Disclose tool-output injection examples that do not provide turnkey instructions for abusing live accounts. Security findings involving production systems go through coordinated private reporting.

13. Threats to validity

Simulator fidelity. A simulator cannot reproduce every matching-engine behavior, venue outage, cross-venue liquidity event, account restriction or internal risk system. A passing simulation is evidence about a defined threat model, not live-production readiness.

Incomplete authorization model. Risk controls often depend on liabilities, collateral, funding, borrow rates, asset concentration and legal restrictions that are out of scope for an initial spot-only implementation. A system could pass all v1 cases yet fail badly in margin or derivatives markets. Its score should never be presented as derivatives safety certification.

Correlated attack generation. Many variants of one prompt-injection strategy may look diverse but share the same semantics. An agent that resists 100 closely related strings may still fail against one new attack class. Statistical analysis must retain attack-family and scenario hierarchy.

Policy/oracle common-mode failure. If the benchmark oracle imports the exact same code as the enforced policy system, the benchmark may validate a shared bug. Independent reference assertions, property tests and human-reviewed golden fixtures mitigate but do not eliminate this risk.

Model-provider drift and hidden serving behavior. Commercial APIs can change routing, system instructions or model implementation. Report configuration as precisely as available and date every run. Exact replication across providers and future dates may remain impossible.

Deployment gap. In production, operator mistakes, key management, access-control misconfiguration, internal compromise, logging failures and network partitioning can defeat safeguards. The benchmark isolates components under controlled assumptions; operational security requires broader review.

14. Artifact package and reproducibility requirements

A research-quality release should include:

crypto-agent-bench/
  CITATION.cff
  LICENSE
  docs/{benchmark-card,security-model,data-card,ethics,limitations}.md
  scenarios/v1.0/{S1,S2,S3,S4,S5,S6,S7}/
  fixtures/v1.0/{books,accounts,instruments,policies,events}/
  src/{runner,model_adapters,tool_adapters,simulator,oracle,metrics}/
  experiments/{baseline,ablation,attack_robustness,recovery}/
  analysis/{notebooks,statistics,bootstrap}/
  reports/{templates,example-synthetic-output}/
  tests/{schema,oracle,golden,property,replay}/
  environment.lock

The benchmark card should state intended use, non-intended uses, dataset origins, venue API assumptions, languages, coverage gaps, known limitations, model cost constraints and how to reproduce a reported score. A publication must cite immutable commits or archived release identifiers, not a moving main branch.

We recommend Apache-2.0 or another permissive software license after checking dependencies, plus clear licensing for synthetic fixtures. Third-party market records should not be redistributed without permission.

15. Proposed release gates

16. Conclusion

The next generation of financial agents will be judged not just on whether they can reason about a chart, but on whether they can act without violating the permissions, limits and operational invariants of the system that deploys them. That is a different measurement problem from financial QA, tool-call syntax, backtest return, or prompt-injection refusal considered in isolation.

Crypto Agent Bench proposes a method for testing this broader question. Its main methodological commitment is to score observable effects and legitimate outcomes separately. It treats the agent as a composed system, tests against authoritative synthetic state, retains exchange-specific API semantics, and reports uncertainty rather than point estimates alone.

If implemented and independently replicated, this protocol could help researchers distinguish safer models from safer surrounding architectures, and help exchange engineering teams decide which components they must verify before placing an AI agent anywhere near real financial execution. At this stage it is a concrete, falsifiable research program, not an empirical conclusion.

References

[1] Islam et al. (2023), FinanceBench: A New Benchmark for Financial Question Answering. https://arxiv.org/abs/2311.11944

[2] Yao et al. (2024), τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. https://arxiv.org/abs/2406.12045

[3] Zhu et al. (2026), FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol. https://arxiv.org/abs/2603.24943

[4] Debenedetti et al. (2024), AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. https://arxiv.org/abs/2406.13352

[5] OWASP (2025), Top 10 for Agentic Applications for 2026. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[6] Xiao et al. (2024), TradingAgents: Multi-Agents LLM Financial Trading Framework. https://arxiv.org/abs/2412.20138

[7] Zhao and Raissi (2026), Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs. https://arxiv.org/abs/2608.11232

[8] Qian et al. (2026-10-21), Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research. https://arxiv.org/abs/2610.30113

[9] Binance Developer Docs, Spot Trading Filters. https://github.com/binance/binance-spot-api-docs/blob/master/filters.md

[10] OKX, API v5 Documentation (trading parameters, demo trading, client order ID semantics). https://www.okx.com/docs-v5/en/

[11] Binance Developer Docs, General REST API Information (execution may be unknown after timeout/5xx). https://developers.binance.com/en/docs/products/spot/rest-api

[12] OpenTelemetry, GenAI semantic conventions (telemetry schema and sensitive data guidance). https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/

[13] NIST, AI Risk Management Framework: Generative AI Profile (2024). https://doi.org/10.6028/NIST.AI.600-1

[14] US SEC, Rule 15c3-5 Market Access FAQ (a specific US securities market access framework, not blanket crypto regulation). https://www.sec.gov/rules-regulations/staff-guidance/trading-markets-frequently-asked-questions/divisionsmarketregfaq-0

Declaration: this paper does not report VenueGuard or Crypto Agent Bench experimental results; it specifies intended experiments and verifiable research design. No venue has endorsed, certified or audited the proposed projects. This is not investment advice.