R&D / Banking AI Evaluation Library / Suggested evaluations
Financial-report numerical reasoning
Specification only — not executed
Can a model answer numeric questions from tables and text in financial reports, with the right units and arithmetic?
Find the figures a question needs in a report, calculate the result, and state the unit, scale and source. No advice and no account actions.
Task boundary
- Trigger
- A user asks a numeric question about a financial report.
- Starts when
- The model receives the question and the report extract with its table and text.
- Ends when
- The model returns a value, its unit and scale, the calculation and the cited figures.
- Expected end state
- The answer, scale and derivation match the reference.
- Human approval
- Not needed for the answer.
Prerequisites
- Report extracts with annotated answers and derivations.
Inputs
- Question: One question in English.
- Report extract: A table and the paragraphs around it.
Outputs
- Answer: A number or a span of text.
- Scale and unit: For example thousands or millions, and the currency or percent.
- Derivation: The figures used and the arithmetic.
Permitted actions
- Read the extract.
- Calculate.
- Return the answer with citations.
Forbidden actions
- Give investment advice.
- Use figures that are not in the extract.
Constraints
- Public company reports only.
What counts as correct
Deterministic outcomes
- The numeric answer matches the reference after scale is applied.
Acceptable alternatives
- Rounding that the question allows.
Judgement calls
- The derivation is easy to follow.
Variations to include
- Answers from the table only.
- Answers that need both table and text.
- Questions about change between years.
- Questions where the scale is stated only in a heading.
Example failures
- Uses the wrong scale, such as thousands for millions.
- Picks a figure from the wrong year or row.
- Gets the arithmetic wrong.
- Gives an answer with no figure from the extract behind it.
Ground truth design
Use the annotated answers, scales and derivations published with the data set.
Evaluation criteria
Definitions only. Dodone has not measured any of these. A safety gate fails the case on a single breach and is never averaged with other criteria.
| Criterion | Family | Metric | Ground truth | Gate |
|---|---|---|---|---|
| Answer correctness | NLP or math metric | Exact match Share of questions where the normalised answer equals the reference answer. | Reference answers from the data set. | No |
| Answer overlap | NLP or math metric | Token-level F1 The F1 score of word overlap between the answer and the reference. | Reference answers from the data set. | No |
| Scale and units | Deterministic check | Scale agreement Share of numeric answers where the stated scale matches the reference scale. | Reference scale labels. | No |
| Citations | Deterministic check | Cited figure check Any figure in the derivation that is not in the extract fails the case. | The report extract. | Safety gate |
Suggested data sources
- TAT-QA Direct fit Licence verified
Why it helps: It has questions over financial report tables and text, with answers, scales and derivations.
What it does not validate: Banking operations, bank-internal reports and regulatory returns.
BIAN v14.0 references
Each service domain below exists in BIAN release 14.0. The link to this evaluation is a Dodone proposal; no service operation is mapped.
- Financial Statement Assessment BIAN v14.0 source entity
Mapping pending verification
Limits
Not tested
- Core banking operations or compliance decisions.
- Bank-internal reports or regulatory returns.
Limitations
- It tests general financial report reasoning, not a banking process.
- Public reports may be in the training data of current models.
Review notes
- The BIAN link is contextual only and is pending verification.
References
Version history
- Version 0.1.0 (this page)