R&D / Banking AI Evaluation Library / Suggested evaluations
Banking customer query intent identification
Specification only — not executed
Can a model classify a banking customer enquiry into the right intent, or abstain when no intent fits?
Read one customer query and return one intent label from a fixed list, or abstain. The model takes no banking action.
Task boundary
- Trigger
- A customer sends a short text query to a banking channel.
- Starts when
- The model receives the query text and the list of permitted intents.
- Ends when
- The model returns one intent label or an abstention.
- Expected end state
- The label matches the reference label, or the model abstains on an out-of-scope query.
- Human approval
- Not needed. The label only routes the query.
Prerequisites
- A fixed, versioned list of intents.
- Labelled queries for each intent.
Inputs
- Customer query: One short text in English.
- Intent list: The permitted intent labels.
Outputs
- Intent: One label from the list, or "abstain".
Permitted actions
- Read the query.
- Return a label or abstain.
Forbidden actions
- Carry out the request in the query.
- Return a label that is not in the list.
Constraints
- No customer account data is used.
What counts as correct
Deterministic outcomes
- The label equals the reference label.
Acceptable alternatives
- Abstention on a query that experts judge ambiguous between two intents.
Judgement calls
None stated.
Variations to include
- Queries with spelling errors.
- Queries that sit between two similar intents.
- Queries outside banking, for testing abstention.
Example failures
- Picks a similar but wrong intent.
- Returns a label that is not in the list.
- Answers out-of-scope queries with a banking intent.
Ground truth design
Use the labels published with the data set. Experts can settle boundary cases. Out-of-scope queries must be added and labelled separately.
Evaluation criteria
Definitions only. Dodone has not measured any of these. A safety gate fails the case on a single breach and is never averaged with other criteria.
| Criterion | Family | Metric | Ground truth | Gate |
|---|---|---|---|---|
| Classification | NLP or math metric | Macro F1 The mean of the F1 score of each intent, so every intent counts equally. | Intent labels from the data set. | No |
| Classification | NLP or math metric | Accuracy for each intent Share of queries for each intent that get the right label. | Intent labels from the data set. | No |
| Confusion | NLP or math metric | Confusion between similar intents The most frequent wrong pairs from the confusion matrix. | Intent labels from the data set. | No |
| Abstention | Deterministic check | Out-of-scope abstention rate Share of out-of-scope queries where the model abstains. | A separate labelled set of out-of-scope queries. | No |
| Safety | Safety gate | Action taken from a query Any attempt to carry out the request fails the case. | The list of forbidden actions. | Safety gate |
Suggested data sources
- BANKING77 Direct fit Licence verified
Why it helps: It has labelled banking customer queries across 77 intents, which matches this task.
What it does not validate: Out-of-scope abstention, other languages and multi-step conversations.
BIAN v14.0 references
Each service domain below exists in BIAN release 14.0. The link to this evaluation is a Dodone proposal; no service operation is mapped.
- Contact Routing BIAN v14.0 source entity
Mapping pending verification - Contact Handler BIAN v14.0 source entity
Mapping pending verification
Limits
Not tested
- An end-to-end customer service conversation.
- Other languages, other banks or voice channels.
Limitations
- Fine-grained intent detection is one component skill, not a full customer service evaluation.
- Results on one data set may not carry over to another bank.
Review notes
- BIAN mappings are candidates only and are pending verification.
References
Version history
- Version 0.1.0 (this page)