Loading rubric…
A BENCHMARK FOR FINANCIAL AGENTS
AgenticFinBench
Benchmarking Financial Agents on Diverse Tasks
from Real-World Requests
Financial work is open-ended. Its evaluation should be grounded.
We pair question-specific rubrics with claim-level auditing to assess
the quality and factual reliability of financial agents.
NashNova Research · Manuscript snapshot · October 2026
230real-world tasks
8task categories
2languages
4complexity levels
18agent configurations
01 / PERFORMANCE
Performance Leaderboard
Quality and generation efficiency.
All 18 configurations are compared on the same 230 questions. Quality Q combines rubric judgments with audit evidence; B is the rubric-only baseline. Rankings use point estimates, so adjacent ranks need not differ significantly.
| Rank | Model | Harness | Q ↑ | 95% CI | Q − B | Rank B → Q | Time ↓ (min) | Cost ↓ (USD) |
|---|---|---|---|---|---|---|---|---|
| Loading frozen manuscript results… | ||||||||
Q 0–100 scaleCI 10,000 paired question-bootstrap resamplesBold best; underlined second and third in Q, time and cost
How to read these results
Q is the quality score; Q − B and B → Q show score and rank changes after auditing. Rows follow mean Q by default. Higher Q is better; lower generation time and cost are better. Bold marks the best value and underlining the second and third best, independent of the selected sort order. Confidence intervals are 95% paired question-bootstrap intervals from 10,000 resamples.
Values match the manuscript main table, rounded as reported. Differences may not equal subtraction of displayed rounded values. Time and cost are generation averages, excluding evaluation cost. Fourteen configurations use recorded-token costs on the 230 questions; Kimi, Trae and DuMate retain original run-average costs, while WorkBuddy retains its visible-text estimate.
The source manifest records figure and table hashes. Download results for additional baseline and hallucination diagnostics.
02 / LOOK INSIDE
Explore the tasks & rubrics
Inspect the original question and its frozen positive and negative criteria. Question text and rubric language are preserved as released.
Loading questions…
03 / THE DATASET
Data Description & Distribution
A cross-section of financial work, as people actually ask for it.
From explaining a financial concept to designing an investment plan, the benchmark spans eight types of deliverable and four levels of information synthesis and decision autonomy.
189Chinese questions
41English questions
Four levels of complexity
L1
Direct retrieval10 questions · 4.3%
L2
Structured synthesis70 questions · 30.4%
L3
Focused analysis86 questions · 37.4%
L4
Research & decision design64 questions · 27.8%
Levels describe task requirements, not measured difficulty.
01
Knowledge Q&A
Concepts, mechanisms, rules, and methods.
49 tasks02
Data & calculation
Retrieve facts and perform specified calculations.
14 tasks03
Screening
Find securities or products matching criteria.
18 tasks04
Comparison
Compare named securities or products.
17 tasks05
Investment research
Assess prospects, value, and investment risks.
62 tasks06
Financial analysis
Interpret disclosures and operating performance.
17 tasks07
Event analysis
Assess the effects of news, policy, and shocks.
38 tasks08
Investment planning
Forecast, allocate, and design strategies.
15 tasks04 / FROM REAL REQUESTS TO A BENCHMARK
Data Construction Process
Real requests. Careful selection. Preserved intent.
Questions originate in production logs of a financial assistant. Three selection batches are filtered, sampled, anonymized, and reviewed to produce a frozen evaluation set.
01
Collect & screen
14,697 query records and 23,731 production traces provide candidates. Filters remove duplicates, unsuitable or out-of-scope queries, and requests dependent on missing context.
02
Sample & anonymize
Seed, expansion, and English batches contribute 44, 146, and 40 final questions. Experts remove personal information while retaining the original intent and phrasing.
03
Review & freeze
A second expert review removes unclear questions. Frozen question-specific rubrics contain 5,817 positive criteria and 1,780 negative criteria.
The benchmark covers selected real requests; it is not a probability sample of production traffic. Batch composition (44 + 146 + 40) differs from language composition (189 Chinese + 41 English).
05 / EVALUATION FRAMEWORK
From frozen inputs to a quality score.
Question-specific rubrics assess the work. A two-stage claim audit checks evidence attribution and factual reliability. Audit evidence then informs the final quality score.
01 / RUBRICS
Did it answer the request?
Positive and negative criteria assess task completion and analytical quality without requiring a single reference answer.
02 / CLAIM AUDIT
What supports the facts?
Claims are checked against the execution trace; ungrounded claims are then checked with external evidence.
03 / QUALITY
How much does an error matter?
Audit evidence affects the quality score according to the scope, severity, and downstream impact of the error.
06 / WHAT THE EXPERIMENTS REVEAL
Key Findings
The answer changes when you check the facts.
01
Factual auditing changes the ranking.
DeepSeek V4.1 Flash on Pi leads under the rubric-only baseline, but loses 7.58 points after the audit and moves to fourth. GPT-5.6 Sol on Codex CLI loses 0.27 points and moves from fifth to first.
−7.58points for Pi
after factual auditing
after factual auditing
Point-estimate ranks; adjacent configurations may not differ significantly.
02
Small claim-level error rates reach many answers.
Claim-level hallucination rates range from 0.03% to 2.38%, yet 0.9%–47.0% of responses contain at least one hallucinated claim. Density and reach tell different stories.
0.9–47.0%responses containing
a hallucinated claim
a hallucinated claim
Hallucination here means ungrounded and externally refuted. It is not a complete factual-error rate.
03
The harness can matter more than the model.
Holding DeepSeek V4.1 Flash fixed, quality spans 17.52 points across Codex CLI, Pi, DSH, and WorkBuddy. Holding Codex CLI fixed, the four GPT models span 10.51 points.
17.52point quality spread
across four harnesses
across four harnesses
These contrasts are observational: instructions, tools, and reasoning settings also vary.
Financial data is associated with stronger grounding.
Across sixteen configurations with complete traces, financial-data use is associated with 26.3 percentage points more grounded claims after controlling for configuration and category. This is an association, not a causal estimate.
Complex tasks expose reliability gaps.
Averaged over configurations, hallucinated responses rise from 12.2% at L1 to 19.6% at L4. Longer, more complex answers expose more claims to error even when rubric scores remain high.
BUILD ON AGENTICFINBENCH
Bring your agent.
Keep the standard.
Use the frozen tasks and rubrics to evaluate your own responses. The standalone runner handles extraction, evidence auditing, and quality scoring.
Read the evaluation guide ↗QUICK START
git clone https://github.com/NashnovaResearch/AgenticFinBench.git
cd AgenticFinBench
cp .env.example .env
# Add your own API key to .env
./run.sh --answers examples/one-answer.jsonl --out results/demoLinux x86_64 · Python 3.11+ · Node.js 22+
The one-question example makes paid API calls.