Pith. sign in

REVIEW 1 cited by

Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-agent LLM system extracts financial KPIs from SEC filings at about 95% accuracy, close to human annotators.

desk verdict Plausible system design, but the headline 95% and human-parity claims are not backed by the paper's own tables; needs artifact release and a real human baseline before it's citable. read the letter →

arxiv 2505.19197 v3 pith:Y4RIITBC submitted 2025-05-25 cs.AI

Chanyeol Choi , Alejandro Lopez-Lira , Yongjae Lee , Jihoon Kwon , Minjae Kim , Juneha Hwang , Minsoo Ha , Chaewoon Kim
show 3 more authors
Jaeseon Ha Suyeol Yun Jin Kim
This is my paper · ORCID
classification cs.AI
keywords FinancialInformationExtractionMulti-AgentSystemsText-to-SQLAnalyticsKeyPerformanceIndicatorsSECfilingsLargeLanguageModels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a modular multi-agent system built from large language models can replace manual KPI extraction in investment research. It claims that splitting the work into an Extraction Agent and a Text-to-SQL Agent turns 1,200 unstructured SEC filings into structured records at roughly 95% accuracy, about the level of trained human annotators, and that 91% of natural-language retrieval answers were judged correct by human evaluators. A sympathetic reader would care because, if this holds, the bottleneck of reading 10-Ks, 10-Qs, and earnings transcripts by hand disappears, and analysts can query the resulting data in plain English.

What carries the argument

The load-bearing mechanism is the Domain Rule Injection stage inside the Extraction Agent, a set of heuristics that normalize numeric ranges such as converting guidance of 22–24% to a 23% midpoint, resolve unit mismatches, disambiguate fiscal periods, and separate actual results from forward-looking guidance and GAAP from non-GAAP figures. Combined with QA-driven validation, this rule layer carries the accuracy gains: removing it raises the unit error rate from 2.3% to 12.9% and period misalignment from 3.1% to 15.4%, while dropping extraction precision from 91.2% to 82.6%.

What would settle it

Hold out a set of filings and analyst queries produced after the rules and prompts were frozen, run both agents, and compare the structured records and query answers against fresh human annotation; if the results fall more than about ten percentage points below the reported 95% extraction and 91% retrieval accuracy, the central generalization claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that decomposing financial document processing into two specialized LLM agents yields human-level structuring accuracy and reliable query answering. The Extraction Agent identifies KPI spans, standardizes values and units, resolves fiscal periods, and verifies outputs; the Text-to-SQL Agent turns analyst questions into validated, executable SQL over the resulting schema. The paper reports that the full pipeline reaches 95.3% metric precision and 91.6% structuring accuracy in extraction, and 91.2% top-1 SQL accuracy in a human evaluation, with rule-based validation responsible for most of the gain over plain LLM prompting.

Load-bearing premise

The load-bearing assumption is that the cleanup rules built into the Extraction Agent were general heuristics and were not tuned on the 1,200 filings and 312 queries used for testing; if those rules came from the test corpus, the reported gains would not transfer to new documents.

Editorial extensions

If this is right

  • If the accuracy claim holds, financial analysts could query structured KPI databases in natural language without knowing the database schema, cutting manual data entry from the workflow.
  • The 2.4× throughput gain suggests that the same analyst team could cover more filings per day, or cover smaller companies that are currently skipped for cost reasons.
  • The modular split means extraction errors and query errors can be diagnosed and fixed independently, which is what makes the reported auditing possible.
  • Document types beyond 10-Ks and 8-Ks, such as earnings releases and transcripts, are claimed to work in the same pipeline, so the method generalizes rather than being tied to one filing form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run is temporal holdout: freeze the rules and prompts, then evaluate on filings published after the freeze date; that would separate genuine generalization from tuning on the test distribution.
  • The rule-injection results suggest the same two-agent decomposition could transfer to other regulated-text domains, such as clinical trial reports or regulatory disclosures, but the specific heuristics would need to be rewritten for each domain's units and period conventions.
  • The 95% accuracy figure is KPI-dependent; the paper's KPI set is analyst-defined, so shifting to a broader or differently defined metric list would likely change the measured ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: empirical system evaluation is self-contained; unsupported 95% parity claim is a verification issue, not circularity.

full rationale

This is an empirical systems paper rather than a formal derivation; there is no equation chain in which a predicted quantity is equal to an input by construction. The Extraction and Text-to-SQL agents are evaluated against manually annotated gold KPI labels (Section 4.1), and the reported metrics (Extraction F1, Metric Precision, Structuring Accuracy, Unit Error Rate, SQL validity, execution success) are standard component measurements against those external labels. The Domain Rule Injection heuristics in Section 3.3 are a pipeline stage whose contribution is assessed by ablation (Tables 1 and 5), not a parameter fitted to the test labels in any stated way; absent a statement that the rules were tuned on the evaluation filings, inferring leakage would be speculation, which the circularity standard forbids. The reference list contains no load-bearing self-citation: no uniqueness theorem, no ansatz adopted solely from the authors' prior work, and no result imported from a same-author paper. The abstract's 'approximately 95% accuracy' does not correspond to a single table line (Table 2 reports Metric Precision 95.3% and Structuring Accuracy 91.6%; Table 1 reports Extraction F1 91.2%), and the 'matching human annotators' claim is not backed by a reported inter-annotator agreement baseline. Those are verification/correctness weaknesses in the headline claims, not circular reductions, so they do not raise the circularity score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper does not derive results from axioms; it reports empirical accuracy. The main unstated inputs are the hand-crafted rules, prompts, and the private evaluation set, which together control the reported numbers.

free parameters (2)
  • Domain rule injection heuristics
    Hand-crafted normalization and validation rules in Section 3.3; not disclosed and possibly tuned on the evaluation data.
  • Prompt templates for extraction and SQL generation
    Domain-aware prompts are hand-designed (Appendix A) and treated as part of the system; their exact text is not fully provided.
assumptions (3)
  • domain assumption The ground-truth annotations by 'trained financial analysts' are accurate and complete.
    Section 4.1 states annotations were made by trained analysts and cross-validated, but no inter-annotator agreement is reported.
  • domain assumption Claude 3.7 and GPT-4 outputs are deterministic enough for the reported measurements to be meaningful.
    No repeated runs or variance estimates are provided.
  • domain assumption The 1,200 SEC filings and 312 queries are representative of the target financial documents.
    Section 4.1 describes the documents as 'simulated real-world analyst workflows,' but selection criteria are not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance." pith.science (2026). https://pith.science/paper/Y4RIITBC

@misc{pith2026250519197,
  author       = {Pith},
  title        = {Pith review of: Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y4RIITBC}},
  note         = {Machine review of arXiv:2505.19197}
}
read the original abstract

Extracting structured and quantitative insights from unstructured financial filings is essential in investment research, yet remains time-consuming and resource-intensive. Conventional approaches in practice rely heavily on labor-intensive manual processes, limiting scalability and delaying the research workflow. In this paper, we propose an efficient and scalable method for accurately extracting quantitative insights from unstructured financial documents, leveraging a multi-agent system composed of large language models. Our proposed multi-agent system consists of two specialized agents: the \emph{Extraction Agent} and the \emph{Text-to-SQL Agent}. The \textit{Extraction Agent} automatically identifies key performance indicators from unstructured financial text, standardizes their formats, and verifies their accuracy. On the other hand, the \textit{Text-to-SQL Agent} generates executable SQL statements from natural language queries, allowing users to access structured data accurately without requiring familiarity with the database schema. Through experiments, we demonstrate that our proposed system effectively transforms unstructured text into structured data accurately and enables precise retrieval of key information. First, we demonstrate that our system achieves approximately 95\% accuracy in transforming financial filings into structured data, matching the performance level typically attained by human annotators. Second, in a human evaluation of the retrieval task -- where natural language queries are used to search information from structured data -- 91\% of the responses were rated as correct by human evaluators. In both evaluations, our system generalizes well across financial document types, consistently delivering reliable performance.

Figures

Figures reproduced from arXiv: 2505.19197 by the authors.

Figure 1
Figure 1. Comparison between the conventional manual pipeline and our proposed agentic system for financial KPI extraction [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Manual KPI extraction process from earnings re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of our multi-agent system for financial document understanding. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AnnoRetrieve: Efficient Structured Retrieval for Unstructured Document Analysis

    cs.IR 2026-04 unverdicted novelty 7.0 of 10

    AnnoRetrieve uses auto-generated structured schemas and queries to retrieve information from unstructured documents more efficiently and accurately than embedding-based methods.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shreyas Agashe, Yifan Fan, Aidan Reyna, and Xiaoyuan Edward Wang. 2023. LLM- Coordination: Evaluating and Analyzing Multi-Agent Coordination Abilities in Table 5: Ablation: Without Rule-Based Validation Metric With Rules Without Rules 𝛿 Error Extraction Precision 91.2% 82.6% -8.6% Unit Error Rate 2.3% 12.9% +10.6% Period Misalignment Rate 3.1% 15.4% +12.3...

    arXiv 2023

  2. [2]

    Mario Costantino and Paolo Coletti. 2008. Information Extraction in Finance. WIT Press

    work page 2008

  3. [3]

    Mario Costantino and R. G. Morgan. 1997. Natural language processing and information extraction: Qualitative analysis of financial news articles. In Proceed- ings of the IEEE International Conference on Systems, Man, and Cybernetics . IEEE, 3870–3875

    work page 1997

  4. [4]

    Gilles Jacobs and Veronique Hoste. 2022. SENTiVENT: enabling supervised information extraction of company-specific events in economic and financial news. Language Resources and Evaluation 56, 4 (2022), 1121–1150

    work page 2022

  5. [5]

    Raghav Kumar, Akshay Dibbu, Shubham Harsola, et al . 2024. BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain. arXiv preprint arXiv:2401.12345 (2024). https://arxiv.org/abs/2401.12345

    arXiv 2024

  6. [6]

    Hong Li, Hao Gao, and Chao Wu. 2024. Extracting financial data from unstruc- tured sources: Leveraging large language models. Journal of Information Systems (2024). https://publications.aaahq.org/jis/article/38/1/129/12345

    work page 2024

  7. [7]

    Xiaoxi Li, Sheng Wang, Shu Zeng, Ying Wu, and Yihan Yang. 2024. A Survey on LLM-Based Multi-Agent Systems: Workflow, Infrastructure, and Challenges. Springer (2024). https://link.springer.com/article/10.1007/s10462-024-10621-2 Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance SIGIR FinIR’25, July 17...

    work page doi:10.1007/s10462-024-10621-2 2024

  8. [8]

    Hammad H Malik, Vishwas S Bhardwaj, and Herb Fiorletta. 2011. Accurate information extraction for quantitative financial events. InProceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1253–1254

    work page 2011

Show all 19 references
  1. [9]

    Sarthak Shah, Sai Ryali, and Raghav Venkatesh. 2024. Multi-Document Financial Question Answering using LLMs. arXiv preprint arXiv:2411.07264 (2024). https: //arxiv.org/abs/2411.07264

    2024 arXiv

  2. [10]

    Mudassar Sheikh and Sumali Conlon. 2012. A rule-based system to extract financial information. Journal of Computer Information Systems 52, 4 (2012), 88–97

    2012

  3. [11]

    Akshay Singh, Aishwarya Shetty, Atif Ehtesham, et al. 2025. A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applica- tions, Use Cases, and Challenges. InProceedings of the 2025 IEEE 15th International Conference on Big Data Science and Enginee...

    2025

  4. [12]

    Yujie Song, Saif Ezzini, Xia Tang, Carl Lothritz, and Janos Klein. 2024. Enhancing Text-to-SQL Translation for Financial System Design. In Proceedings of the 46th ACM Symposium on Applied Computing . ACM

    2024

  5. [13]

    Prashant Srivastava, Meghna Malik, Varun Gupta, Tapan Ganu, et al. 2024. Evalu- ating LLMs’ Mathematical Reasoning in Financial Document Question Answering. arXiv preprint arXiv:2402.13555 (2024). https://arxiv.org/abs/2402.13555

    2024 arXiv

  6. [14]

    Khanh-Tung Tran, Duy Dao, Minh-Duy Nguyen, Quoc-Viet Pham, et al. 2025. Multi-Agent Collaboration Mechanisms: A Survey of LLMs. arXiv preprint arXiv:2501.06322 (2025). https://arxiv.org/abs/2501.06322

    2025 arXiv

  7. [15]

    Niklas Wretblad and Fredrik Gordh Riseby. 2024. Bridging Language and Data: Optimizing Text-to-SQL Generation in Large Language Models. DIV A Portal (2024). https://www.diva-portal.org/smash/get/diva2:1823187/FULLTEXT01.pdf

    2024

  8. [16]

    Xinqi Yang, Scott Zang, Yong Ren, Dingjie Peng, and Zheng Wen. 2024. Evaluating Large Language Models on Financial Report Summarization: An Empirical Study. arXiv preprint arXiv:2411.06852 (2024). https://arxiv.org/abs/2411.06852

    2024 arXiv

  9. [17]

    Yujing Yang, Haotian Chai, Shu Shao, Yihan Song, Shan Qi, Ruijie Rui, et al. 2025. AgentNet: Decentralized Evolutionary Coordination for LLM-Based Multi-Agent Systems. arXiv preprint arXiv:2503.10777 (2025). https://arxiv.org/abs/2503.10777

    2025 arXiv

  10. [18]

    Cheng Zhang, Yifan Mao, Yu Fan, Yutong Mi, Yu Gao, and Ling Chen. 2024. Fin- SQL: Model-Agnostic LLMs-Based Text-to-SQL Framework for Financial Analysis. Companion of the 2024 ACM International Conference on AI in Finance (2024)

    2024

  11. [19]

    Guilherme Gomes Ziegler. 2024. Automating Information Extraction from Financial Reports Using LLMs. Aalto University Publications (2024). https: //aaltodoc.aalto.fi/handle/123456789/123456

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.