Skip to content

buaacxf/FinanceComplexQA

BranchesTags

Repository files navigation

Finance-ComplexQA

A bilingual Chinese-English benchmark for document-grounded complex financial QA

License Hugging Face

📖 About · 🚀 Quick Start · 📊 Overview · 🗂️ Schema · 🏗️ Construction · 📝 Citation

Finance-ComplexQA at a Glance

📖 About

Finance-ComplexQA evaluates whether LLMs and agents can answer complex finance questions by reasoning over full reference documents — not just retrieving a single passage from parametric knowledge.

What it tests: multi-hop reasoning, numerical calculation, comparison, implicit inference, planning, summarization, and evidence-grounded verification across financial corpora.

Three components:

Component Purpose
FinComplexQA-Pro Primary benchmark — 2,026 unique QA pairs, organized by language, financial scene, and task category
overall Extended set with agent outputs (agent_answer, agent_thinking) and LLM-as-a-judge scores
Reference_documents Source PDF + LaTeX documents that ground each question

Each QA example in FinComplexQA-Pro appears in two views (scene_categories and task_categories), yielding 4,052 record views total. FinComplexQA-Pro and overall serve different purposes and are not disjoint.

🚀 Quick Start

Because the repository contains heterogeneous schemas, users may prefer to load selected subdirectories rather than merge all JSONL files at once.

Download the Dataset Snapshot

from huggingface_hub import snapshot_download

repo_dir = snapshot_download(
    repo_id="Multilingual-Multimodal-NLP/FinanceComplexQA",
    repo_type="dataset"
)
print(repo_dir)

Read JSONL Files Manually

import json
from pathlib import Path

root = Path(repo_dir)
path = root / "FinComplexQA-Pro" / "EN" / "task_categories" / "Numerical_Comparison"

examples = []
for file in path.glob("*.jsonl"):
    with file.open("r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                examples.append(json.loads(line))

print(len(examples))
print(examples[0])

Load a Homogeneous Subset with datasets

from pathlib import Path
from datasets import load_dataset

data_files = {
    "train": str(
        Path(repo_dir)
        / "FinComplexQA-Pro"
        / "EN"
        / "task_categories"
        / "Numerical_Comparison"
        / "*.jsonl"
    )
}

dataset = load_dataset("json", data_files=data_files)
print(dataset)
print(dataset["train"][0])

Evaluate Model Outputs

Finance-ComplexQA can support several evaluation settings:

  1. Answer accuracy against the gold answer, using exact match, numerical tolerance, F1, semantic similarity, or task-specific metrics.
  2. Evidence completeness and evidence coverage, especially for long-form answers.
  3. Faithfulness to the reference document.
  4. Reasoning quality, using available reasoning, sub_questions, sub_answers, or agent_thinking fields.
  5. LLM-as-a-judge calibration, using the judge outputs in overall.
  6. Agent-as-a-judge evaluation for accuracy, faithfulness, and evidence coverage.

📊 Overview

Count
Unique QA examples (FinComplexQA-Pro) 2,026
— Chinese / English 1,013 / 1,013
Record views (scene + task views) 4,052
Agent-evaluated examples (overall) 3,140
Reference documents (PDF + LaTeX pairs) 1,041 each

📁 Repository Structure

Finance-ComplexQA/
├── FinComplexQA-Pro/
│   ├── CN/  {scene_categories, task_categories}/
│   └── EN/  {scene_categories, task_categories}/
├── overall/                  # agent outputs + judge scores
├── Reference_documents/
│   ├── CN/
│   └── EN/
└── Illustration/

🏢 Scene Categories (scene_categories)

Scene Category Lang Examples Description
Bank_Financial_Statements CN 139 Bank financial statements
Corporate_Financial_Report CN 163 Chinese corporate financial reports
Customer_Service_Records CN 140 Customer service & transaction records
FinTech_Research_Report CN 134 FinTech industry research
Investment_Strategy_Report CN 246 Investment strategy & market analysis
Market_Trend_Analysis_Report CN 91 Market trend & macro/industry analysis
Supervision_and_Compliance_Audit CN 100 Supervision, compliance & audit
Corporate_Financial_Report EN 253 International corporate financial reports
FinTech_Research_Report EN 157 Global FinTech research
Government_Fiscal_Bulletin EN 135 Government fiscal bulletins
Investment_Strategy_Report EN 276 Cross-market investment strategy
Market_Trend_Analysis_Report EN 192 Global market trend analysis

🧠 Task Categories (task_categories)

Task Category Lang Examples Description
Comparison CN 190 Compare numbers, policies, entities, or events
Implicit_Reasoning CN 163 Infer unstated financial relationships
Knowledge_Query CN 141 Retrieve domain knowledge from documents
MultiHop_Reasoning CN 228 Connect multiple knowledge points
Planning CN 137 Analysis plans under stated assumptions
Summary CN 154 Integrate scattered evidence
Explicit_Reasoning EN 195 Reason over explicitly stated facts
Implicit_Reasoning EN 220 Infer implicit relationships
MultiHop_Judgment EN 141 Multi-step reasoning + calculation + judgment
Numerical_Comparison EN 113 Numerical comparison & calculation
Planning EN 101 Financial analysis plans
Summary EN 243 Summarize scattered reference information

Task categories and example questions

📈 Extended Set: overall

For analyzing model behavior and judge calibration. Subsets include 公司财报-中文, 国内公司混合-中文, 国际公司混合-英文, 投资策略报告-中文/英文, 政府财政公报-英文 (3,140 examples total).

Extra fields: agent_thinking, agent_answer, llm-as-a-judge, code_stdout/stderr/returncode, sub_questions/sub_answers.

🗂️ Data Schema

Schemas vary by task type — always inspect your chosen subset before use.

Core fields (all subsets):

Field Description
finqa_id Unique example ID
question Financial question
gold Reference answer
Reference_documents Path to supporting PDF/LaTeX source
doc_type Document scene category
task_type Reasoning task category
company_with_period Company, institution, or fiscal period

Optional fields (task-dependent): external_knowledge, decomposed_queries, related_questions, reasoning, sub_questions, sub_answers.

🧩 Examples

CN — Comparison

{
  "finqa_id": "indCN_01269",
  "question": "请对比企业合并和外币货币性项目的会计处理方法有哪些不同？",
  "doc_type": "Bank_Financial_Statements",
  "Reference_documents": "outputs/三川智慧科技股份有限公司/...tex",
  "gold": ["企业合并的会计处理方法依据是否同一控制下分为两种..."],
  "task_type": "Comparison"
}

EN — Numerical Comparison

{
  "finqa_id": "indEN_00673",
  "question": "Using each company's FY2015 financial statements, compare Boeing, CVS Health, Intel, Microsoft, and Apple by calculating their FY2015 dividend payout ratios...",
  "task_type": "Numerical_Comparison",
  "gold": "The highest ratio exceeds the lowest by 59.00 percentage points.",
  "doc_type": "Corporate_Financial_Report"
}

Example with Agent Evaluation

{
  "finqa_id": "finreport_00001",
  "company_with_period": "万泰生物",
  "question": "假如万泰生物2024年的营业利润提高50%,则2022年到2024年三年间的营业利润复合增长率是多少?",
  "Reference_documents": "outputs/financial_reports/万泰生物/万泰生物_企业财务报告_经营绩效与行业比较分析.tex",
  "gold": -0.6075,
  "agent_answer": "-0.8414"
}

🏗️ Construction

Benchmark construction pipeline

The construction pipeline includes:

  1. corpus collection from expert-level financial data sources;
  2. evidence extraction and classification;
  3. LaTeX-based document generation with structured layouts and paragraph evidence;
  4. question generation around document evidence and target reasoning abilities;
  5. web search and MCP-assisted verification where applicable;
  6. check-and-correct refinement using large language models and human verification;
  7. final data collection into Finance-ComplexQA.

Quality control includes independent testing, cross evaluation, expert validation, criteria verification, and multi-source provision. Q&A pairs are checked against two core criteria:

  • answers should be consistent with the questions and reference documents;
  • questions should be solvable based on the reference documents.

⚠️ Limitations

  • For research and evaluation only — not investment, accounting, legal, or financial advice.
  • Financial facts are time-sensitive; verify source documents before real-world use.
  • CN and EN subsets cover different document domains; schemas differ across tasks.
  • llm-as-a-judge scores are diagnostic signals, not ground truth.

Personal and Sensitive Information

The dataset is designed around financial documents and institutional/company-level information. Users should still review the source documents and downstream use cases to ensure compliance with applicable privacy, data redistribution, and financial information regulations.

📜 License · 📝 Citation · 📬 Contact

Released under Apache License 2.0.

@misc{financecomplexqa2026,
  title        = {Finance-ComplexQA: A Bilingual Benchmark for Complex Financial Question Answering},
  author       = {BuaaCXF},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/BuaaCXF/Finance-ComplexQA}},
  note         = {Hugging Face dataset}
}

@misc{cheng2026financecomplexqabenchmarkingagenticreasoning,
      title={FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents}, 
      author={Xianfu Cheng and Shiwei Zhang and Jiyu Zhao and Jian Yang and Xinyuan Wang and Ming Zhou and Weixiao Zhou and Xiangyuan Guan and Xiang Li and Zhenhe Wu and Ziyi Ni and Zhoujun Li and Bingjing Xu},
      year={2026},
      eprint={2607.19238},
      archivePrefix={arXiv},
      primaryClass={cs.CE},
      url={https://arxiv.org/abs/2607.19238}, 
}

Questions and discussions: Hugging Face dataset page

About

A bilingual Chinese-English benchmark for document-grounded complex financial QA

Resources

Readme
Apache-2.0 license
Activity

Stars

5 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages