{"id":"5d628648-74f9-48f2-8d69-278816979338","arxiv_id":"2607.19238","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FinanceComplexQA: 2,026 bilingual expert questions over 1,009 layout-rich financial documents, where top agentic/RAG systems score 55–76, showing large gaps in numerical and cross-layout reasoning.","lead":"This paper introduces FinanceComplexQA, a bilingual benchmark with 2,026 expert-level financial questions over 1,009 long, layout-rich documents, and uses it to evaluate RAG and agentic AI systems. It matters because it tests whether AI assistants can perform real financial analysis—reading tables, paragraphs, and notes together—rather than simple fact retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference answers and judge are unvalidated: the paper's own human evaluation reports 0.0/2.0 correctness-completeness, so reported system rankings may not measure financial reasoning.","rationale":"The benchmark's central claim has two parts: FinanceComplexQA measures realistic, expert-level financial reasoning, and current systems fail at that reasoning. Both rest on the evaluation instrument. If reference answers are not correct/complete, or the judge does not track expert quality, the absolute scores and rankings are uninterpretable. The paper itself provides the strongest evidence: Table 9 reports human reviewers scoring two sampled subsets at 0.0 and 2.0; the text calls for 'more careful reference-answer calibration'. The construction pipeline compounds this concern: B2/B3 create merged questions and reference answers from sub-answers while instructing the model not to question their correctness, and Part A generates the documents from those QA triples. There is no independent, expert-derived ground truth for most of the 2,026 items. The judge prompt, evaluation code, and dataset are not released, so the reported numbers cannot be reproduced or audited. This is not a 'different consensus' issue; it is an internal-evidence problem: the paper's own human evaluation contradicts its claim of 'relatively stable and permanent reference answers'. A concrete human-calibration study on a stratified sample, with the judge applied to the same outputs, would settle whether the rankings survive expert audit. Until then, the appropriate position is conditional: the benchmark is promising but its central claim is unverified. The reader's verdict already reflects this, so no change is needed.","tokens_in":26443,"tokens_out":6982,"duration_ms":69874,"concrete_test":"After releasing the 1,009 documents, 2,026 reference answers, and the exact GPT-5-mini judge prompt, draw a stratified sample of 100 items (oversampling English GFB and the mixed Chinese subset from Table 9). Have three financial analysts independently answer each item and rate the reference answers for correctness/completeness; then compute (i) analyst–reference agreement and (ii) judge–expert rank correlation on the same model outputs. If analyst agreement with the reference answers is not substantially above chance, or judge–expert rank correlation is below 0.7, the reported leaderboard is not a valid measure of financial reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is validity of ground truth and scoring. Reference answers are produced by LLMs: Appendix B2 merges 2–4 sub-questions into a merged question, B3 solves it from sub-answers and a reasoning trace with the explicit instruction 'do not question the correctness of the sub-question answers', and Part A generates the LaTeX document from the QA triple so the answer is derivable. The judge is another LLM (GPT-5-mini), with no reported calibration against expert scores. Section 5.2, Table 9, is the only human check: two participant groups score 0.0 and 2.0 on correctness/completeness. The authors attribute this to needing 'more careful reference-answer calibration' and do not report whether the judge agrees with humans on those subsets. Consequently the claim of 'relatively stable and permanent reference answers' and the system rankings in Tables 5–7 are unsupported; low model scores could reflect judge/annotation artifacts rather than genuine failures in financial reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Finance-LaTeX SKILL, an agent workflow for synthesizing layout-rich financial documents and QA pairs, and FinanceComplexQA, a bilingual benchmark of 2,026 open-ended 'deep research' tasks over 1,009 financial documents. The benchmark is designed to require dual-context reasoning and cross-layout evidence aggregation, and it is evaluated with an Agent-as-a-Judge protocol (GPT-5-mini) using accuracy, ROUGE-style overlap, faithfulness, and coverage metrics. The authors evaluate two RAG/indexing systems and two agentic frameworks across seven LLMs, report that no system exceeds roughly 76 on Chinese or 70 on English scene averages, and provide a failure taxonomy. The paper argues that current agents remain inadequate for long-chain numerical reasoning, cross-layout evidence fusion, and analytical synthesis in finance.","tokens_in":26610,"tokens_out":4364,"duration_ms":48232,"significance":"If the reference answers and judge protocol are valid, FinanceComplexQA would be a valuable resource: it combines bilingual coverage, long documents, complex layouts, and open-ended analytical answers, and its construction pipeline is unusually detailed. The failure taxonomy and the explicit comparison of retrieval, page-indexed, and agentic systems are useful contributions. The paper also states its limitations candidly. However, the central claim of 'relatively stable and permanent reference answers' and the reliability of the reported system rankings are not currently supported: the reference answers are produced by an LLM pipeline that is instructed not to question sub-answer correctness, the judge is an LLM from the same model family as several evaluated systems, and the only human evaluation in the paper reports two participant groups with correctness/completeness scores of 0.0 and 2.0. These issues are load-bearing because they affect every number in Tables 5–7.","major_comments":[{"comment":"The human evaluation is the only external validation of the benchmark's ground truth, and it directly undermines the 'stable and permanent reference answers' claim. Participant 6 scores 0.0 and Participant 2 scores 2.0 on correctness-and-completeness. The paper attributes these to needing 'more careful reference-answer calibration,' but it does not report whether the affected samples were revised, removed, or re-annotated, nor does it report whether the GPT-5-mini judge agrees with the human scores on the same 300 QA pairs. Without judge–human agreement, the low scores may indicate that a substantial fraction of reference answers are wrong or incomplete, in which case the system rankings in Tables 5–7 do not measure financial reasoning.","section":"§5.2, Table 9"},{"comment":"The reference-answer construction is self-referential. B2 merges 2–4 sub-questions into a merged question, B3 solves the merged question from sub-answers and a reasoning trace with the explicit instruction 'do not question the correctness of the sub-question answers,' and Part A then generates the document from the QA triple so that the answer is derivable. This means reference answers inherit any errors in the source benchmark answers and in the LLM expansion process, and the document is written after the answer exists. The paper needs external validation of the references—for example, human re-annotation of a stratified sample with an agreement statistic—before the benchmark can be considered a reliable ground truth.","section":"Appendix B2/B3"},{"comment":"The judge is GPT-5-mini, an LLM from the same family as several evaluated models (e.g., GPT-5.4/GPT-5.5 in the closed-source rows). The primary metric ACC is defined as whether the 'final conclusion matches the reference answer,' but this judgment is made by the same LLM judge with no reported calibration against expert scores, no inter-judge reliability, and no analysis of judge bias by model family or by output length. Since Tables 5–7 rely entirely on this judge, the paper should report judge–human agreement on the Table 9 subsets, and ideally include a second independent judge or a rubric-based human scoring of a random sample.","section":"§3.7, Agent-as-a-Judge"},{"comment":"The human evaluation sample is small and the treatment of low-scoring groups is not described. Six participants each review 50 QA pairs, and for the two groups with scores 0.0 and 2.0 the paper does not say whether those QA pairs were later dropped or corrected, or whether the errors were in the reference answers, the documents, or the participant's annotation process. Without this information, the quality-control claims in §3.8 and the 'relatively stable and permanent reference answers' assertion in the abstract are not verifiable. The authors should either demonstrate that the problematic subsets were fixed and re-evaluated, or report the benchmark with those subsets excluded.","section":"§5.2, Table 9"}],"minor_comments":[{"comment":"Typo: 'Chinsne' should be 'Chinese.' Also, the caption lists counts joined by '+' but does not clearly explain which side is which in the table body; a short note in the caption would help.","section":"Table 2"},{"comment":"The abstract says the benchmark has '8 key features' but then lists six. The Introduction also lists 'four core design principles.' This inconsistency should be corrected.","section":"Abstract/Introduction"},{"comment":"The caption says 'Representative task-level results from the draft experiments.' The word 'draft' appears out of place and should be removed or replaced with 'our experiments.'","section":"Table 7 caption"},{"comment":"The ROUGE-style overlap metric is not formally defined (e.g., tokenization, stemming, aggregation over references). Since ROU is described only as a 'diagnostic,' a brief definition or pointer to the implementation would improve reproducibility.","section":"§4.2"},{"comment":"Some references are incomplete or inconsistent, e.g., 'Revanth Gangi Reddy and 1 others. 2024' omits co-authors, and the Databricks OfficeQA-Pro reference is given as an arXiv URL without a full citation. Please standardize the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark has clear potential, but the lack of validated ground truth is the central issue. The paper should not be accepted until the authors provide judge–human agreement data, resolve the low-scoring human-evaluation subsets, and demonstrate reference-answer quality on a stratified sample. I would also encourage releasing the human-evaluated subsets and judge prompts with the revised manuscript; this would substantially increase the benchmark's credibility. The Table 9 results are not a minor footnote—they are the only direct evidence about reference-answer correctness, and they currently contradict the paper's central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a serious referee, but as submitted its central claim—stable, permanent reference answers—is not supported. The construction pipeline is the most detailed I've seen for this kind of benchmark, and the combination of bilingual coverage, cross-layout evidence, and open-ended analytical questions is genuinely new. If the dataset and judge code were released with a credible human-calibration study, this would become a valuable evaluation asset. Don't desk-reject; send it for review with clear revision demands.\n\nWhat's new: Table 1 is honest — no prior benchmark combines long-document parsing, cross-layout reasoning, bilingual scope, and open-ended answers. The 2,026 tasks over 1,009 documents, plus the failure taxonomy (numeric drift, evidence omission, layout confusion, over-synthesis, weak planning), are useful. The finding that PageIndex matches or beats agentic systems at lower cost is a real result.\n\nSoft spots, in order of severity:\n\n1. Ground truth and judge validity. Reference answers are assembled by LLMs (B2/B3) from sub-answers taken from prior benchmarks, with the explicit instruction 'do not question the correctness of the sub-question answers.' The documents are then generated to make those answers derivable. So the benchmark risks being a closed loop: systems are tested on whether they can recover answers that were reverse-engineered into synthetic documents. The paper's own Table 9 is the only human check, and two of six participant groups score 0 and 2 out of 100 on correctness/completeness. The authors attribute this to needing 'more careful reference-answer calibration,' which is accurate but understates the problem: those numbers suggest the reference answers may simply be wrong for those domains, and the judge (GPT-5-mini, same model family as evaluated systems) has no demonstrated agreement with humans.\n\n2. Reproducibility. No dataset, code, or judge prompts are released. Reading the appendix lets you reconstruct the pipeline, but you cannot reproduce a single score. For a benchmark paper, that is disqualifying as submitted.\n\n3. Experimental rigor. Table 7 is explicitly labeled 'draft experiments.' No error bars or significance tests appear anywhere. Given the judge is an LLM, score differences of a few points (e.g., 60.44 vs 59.98) are likely noise.\n\nThe Limitations section does concede judge bias, which is honest. But the paper then proceeds to treat the rankings as solid conclusions.\n\nWho this is for: anyone building or benchmarking financial RAG/agentic systems. The failure taxonomy and the layout-preserving retrieval result deserve attention.\n\nBottom line: a promising artifact, not yet a verified instrument. I'd recommend sending it to peer review with a hard requirement to release artifacts and provide human calibration of both reference answers and judge. If they do that, it could become an important benchmark; if they can't, the headline results are unsupported.","headline":"Well-built benchmark with a genuinely new combination of bilingual, cross-layout, open-ended financial QA, but unvalidated LLM-generated reference answers and judge undermine the reported rankings until artifacts and human calibration are provided.","tokens_in":27199,"tokens_out":2057,"would_cite":false,"duration_ms":23204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinanceComplexQA is a new bilingual benchmark claiming that current AI systems still fail at the core demands of financial document reasoning—long-chain numerical work, cross-layout evidence fusion, and industry-level synthesis.","keywords":["FinanceComplexQA","financial document QA","agentic reasoning","benchmark","cross-layout reasoning","dual-context reasoning","RAG evaluation","bilingual"],"falsifier":"Rescore a random sample of 200 benchmark questions with independent financial analysts who do not see the reference answers, and compare their verdicts with the LLM judge. If expert acceptance of an answer correlates poorly with judge scores, or if experts flag reference answers as incorrect or incomplete, the benchmark's evaluation validity fails.","tokens_in":26270,"feed_emoji":"📊","tokens_out":5736,"duration_ms":54534,"temperature":0.7,"pith_summary":"The paper introduces FinanceComplexQA, a bilingual (Chinese/English) benchmark of 2,026 expert-level research tasks grounded in 1,009 industrial financial documents. Its central claim is that existing RAG and agentic systems, despite strong progress, still fall short of professional-level financial analysis: the best scores reach roughly 76 on Chinese scenes and 70 on English scenes, with many task accuracies below 60. The benchmark is designed around dual-context reasoning—combining explicit document evidence with implicit financial domain knowledge—and cross-layout reasoning that forces answers to integrate paragraphs, tables, forms, and captions. If the benchmark measures what it claims, it exposes a real gap between what retrieval-augmented and agentic models can do and what financial research actually demands, motivating layout-preserving retrieval, explicit evidence planning, and cost-aware routing.","feed_headline":"Best AI agents score below 76 on new financial document benchmark","feed_subtitle":"2,026 expert tasks show systems struggle with long-chain math and cross-layout evidence in real-world reports.","key_machinery":"The load-bearing mechanism is the benchmark's dual-context reasoning design: every question requires explicit document evidence from parsed units (paragraphs, tables, forms, captions) plus implicit financial domain knowledge such as accounting relations, margin interpretation, or regulatory background. Cross-layout reasoning is enforced by construction, forcing systems to aggregate evidence across heterogeneous elements. The benchmark is built with an agent workflow (Finance-LaTeX SKILL) that synthesizes layout-rich financial documents and verified QA pairs, and evaluation uses an Agent-as-a-Judge protocol with multiple metrics (accuracy, numeric correctness, evidence coverage, faithfulness,","core_discovery":"FinanceComplexQA is designed to test whether an agent can perform the full reasoning loop expected in financial analysis: retrieve evidence, preserve layout context, apply domain knowledge, compute or compare quantities, synthesize, and stay faithful. The paper's central finding is that current systems do not pass this test. The best configuration reaches 76.01 on Chinese scenes and 69.39 on English scenes, task-level accuracies mostly sit below 60, and every system shows distinct weaknesses—retrieval systems miss cross-layout evidence, agentic systems hallucinate or omit mandatory planning points. The paper argues that accuracy, faithfulness, and coverage are independent, and lexical overla","pith_inferences":["The benchmark's validity rests on the correctness of LLM-generated reference answers and the LLM judge; the paper's own human evaluation showed two groups scoring near zero, suggesting reference-answer calibration is not yet stable.","The dual-context design could transfer to other expert domains—legal, medical, or regulatory—where documents combine narrative, tables, and implicit professional knowledge.","A testable extension: ablating layout metadata (removing table headers and captions) should measurably drop performance, directly confirming the paper's claim about cross-layout evidence.","The paper's emphasis on stable, verifiable facts limits coverage of live financial reasoning; a forward-looking benchmark would need to incorporate time-sensitive events without sacrificing answer reliability."],"forward_implications":["If correct, the benchmark shows that layout-preserving retrieval—keeping table headers, captions, and page context—is worth substantial performance gains over chunk-based indexing.","Agentic systems improve some open-ended and implicit reasoning tasks but at much higher token and latency cost, implying cost-aware routing is needed for production financial assistants.","The results imply that no single system handles all financial tasks, so benchmarks must separate retrieval accuracy, reasoning accuracy, groundedness, and completeness rather than reporting one aggregate.","The failure taxonomy (numeric drift, evidence omission, layout confusion, over-synthesis, weak planning) provides concrete targets for improving financial AI agents.","The synthetic generation pipeline offers a scalable path for expanding the benchmark to new domains and languages."],"fun_headline_variants":["AI agents flunk new financial doc benchmark, best score 76","New benchmark: AI agents lag in financial reasoning, top 76","FinanceComplexQA: agents fail real-world financial doc tests","Best AI agent hits 76% on tough financial QA, still falls short","Agents struggle with financial reports: new benchmark exposes gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reference answers and the LLM judge must constitute valid ground truth; the paper's own human evaluation, where two groups scored 0.0 and 2.0, suggests the reference answers need more careful calibration.","fun_headline_variants_meta":{"raw":{"variants":["AI agents flunk new financial doc benchmark, best score 76","New benchmark: AI agents lag in financial reasoning, top 76","FinanceComplexQA: agents fail real-world financial doc tests","Best AI agent hits 76% on tough financial QA, still falls short","Agents struggle with financial reports: new benchmark exposes gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1027,"prompt_tokens":752,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":496,"tokens_out":275,"duration_ms":3549,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:59:12.565760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rescore a random sample of 200 benchmark questions with independent financial analysts who do not see the reference answers, and compare their verdicts with the LLM judge. If expert acceptance of an answer correlates poorly with judge scores, or if experts flag reference answers as incorrect or incomplete, the benchmark's evaluation validity fails.","supporting_citations":[],"review_version":1}