R&D experiment. It can change or be removed at any time and is not reviewed or approved. Do not rely on it for decisions. Read the full warning.
R&D / Banking AI Evaluation Library
Suggested evaluations
Specifications for testing an agent, a model or a retrieval step on one banking task.
6 of 6 evaluations
DODONE-BEVAL-001 · version 0.1.0
Can a model or agent tell whether a payment message meets its structural rules, and explain each invalid or missing field correctly?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Model, Agent · Suggested data: ISO 20022 message catalogue (Background only); BIAN semantic APIs (release 14.0.0) (Background only)
DODONE-BEVAL-002 · version 0.1.0
Can an agent find the likely cause of a payment exception, cite the supporting records and policy, and recommend a permitted next action?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Agent, Tool path (trajectory) · Suggested data: ISO 20022 message catalogue (Background only); PaySim synthetic mobile money transactions (Diagnostic only)
DODONE-BEVAL-003 · version 0.1.0
Can a model classify a banking customer enquiry into the right intent, or abstain when no intent fits?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Model · Suggested data: BANKING77 (Direct fit)
DODONE-BEVAL-004 · version 0.1.0
Can an agent find the bank policy that applies, answer accurately from it, and avoid unsupported claims or actions?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Agent, Retrieval, Tool path (trajectory) · Suggested data: τ-bench banking_knowledge domain (Direct fit)
DODONE-BEVAL-005 · version 0.1.0
Can an agent find unusual transaction patterns, collect the evidence and recommend an investigative next step, without claiming that anyone is guilty?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Agent, Model · Suggested data: AMLworld (IBM synthetic AML transactions) (Diagnostic only); PaySim synthetic mobile money transactions (Diagnostic only)
DODONE-BEVAL-006 · version 0.1.0
Can a model answer numeric questions from tables and text in financial reports, with the right units and arithmetic?
Specification only — not executedProposed — not expert reviewedMapping pending verification
Tests: Model, Retrieval · Suggested data: TAT-QA (Direct fit)