{"id":"83fe7a45-3a13-4086-b8be-9c4f4b87d87d","arxiv_id":"2607.18102","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FinSAgent improves financial filing QA by conditioning sub-queries on a summary of the local corpus and gating semantic reranking with a learned validity signal, beating baseline systems on five benchmarks.","lead":"This paper introduces FinSAgent, a multi-agent search-and-answer system for U.S. SEC filings that splits questions by role (quantitative, market, legal, company) and adds a learned 'gate' that demotes boilerplate chunks that semantically match but are not valid evidence. The authors report better answers and retrieval coverage than baseline RAG and multi-agent systems on five financial QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core mechanism attribution is not isolated: the authors admit their evidence rests on cumulative ablations, so DB-aware decomposition and feature-gated reranking may not be the active ingredients.","rationale":"The paper's empirical program is unusually thorough in many respects: five benchmarks, a matched-budget fairness study, a blind human check on SECQUE, long-context comparison, and RAGAS noise sensitivity. I found no evidence of fraud or result fabrication. However, the central conceptual claim—that the two novel mechanisms are what produce the gains—is not yet isolated from cheaper, simpler alternatives. This is not an external-consensus objection; it is an internal one, because the Limitations section explicitly says the evidence 'currently rests on full-system ablations and fixed-pool comparisons' and lists the separated baselines as future work. The cumulative ablation in Table 3 shows inconsistent marginal effects (e.g., gate no-op on FinanceBench, full system below +DB-Aware on FinanceBench), so order effects are not resolved. The reader's weakest assumption about the feature-gate training labels is a specific instance of this broader attribution problem, but the broader problem also covers the query-side mechanism and is more load-bearing. Therefore I would keep the CONDITIONAL verdict, adding a specific condition: demonstrate the two mechanisms against the separated baselines the authors themselves propose.","tokens_in":24294,"tokens_out":12887,"duration_ms":124076,"concrete_test":"Run the two separated baselines the authors say are needed (Limitations): (1) replace the retrieved summary view in §3.3 with a fixed, static corpus-overview prompt, and (2) replace the LightGBM gate in §3.4 with a claim/NLI-style verifier, holding all other components and budgets fixed on all five benchmarks. If either replacement matches FinSAgent's Correctness and Macro-Recall within noise, the specific mechanisms are not the active ingredient; if both degrade, the mechanism claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FinSAgent's gains come from two specific corpus-alignment mechanisms: database-aware query decomposition and feature-gated reranking. But the evidence for this attribution is weaker than the headline suggests. Table 3 is a cumulative ablation, adding components in one fixed order; it does not include either mechanism in isolation or leave-one-out. The authors concede this in the Limitations section: \"Our evidence for the two mechanisms currently rests on full-system ablations and fixed-pool comparisons,\" and they only plan to add \"direct separated baselines\" (static corpus-overview prompt vs. database-aware decomposition; NLI/claim verifier vs. feature-gated reranker). Because these separated baselines are absent, the observed gains could be explained by generic extra context or a generic reranking signal rather than by the proposed principle of corpus-side conditioning. The FinanceBench result in Table 3 (full system 76 vs. +DB-Aware 80) shows an order/interference effect, further indicating that cumulative contributions do not identify stable mechanism-level effects.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinSAgent, a multi-agent retrieval-augmented framework for evidence-grounded question answering over SEC filings. Its central claim is that SEC filing QA is hampered by 'prior–corpus misalignment': model priors cause query decomposition to miss corpus-specific evidence and semantic reranking to favor topically similar but evidentially invalid chunks. FinSAgent instantiates three mechanisms: role-specialized agents anchored to the 10-K item structure, database-aware query decomposition conditioned on a lightweight summary-level corpus view, and multi-path retrieval with a learned feature-gated reranker that combines semantic scores with non-semantic validity features. The paper evaluates on five benchmarks (Lotus, Zeekr, FinanceBench, FinDER, and a 100-question SECQUE subset), reporting improvements in retrieval Macro-Recall and answer correctness over single-agent and multi-agent baselines, plus ablations, a matched-budget fairness study, a blind human validation, a full-document long-context comparison, RAGAS noise sensitivity, overhead measurements, and an error analysis.","tokens_in":24563,"tokens_out":3039,"duration_ms":39351,"significance":"The paper addresses a practical and timely problem: answering complex financial questions over long, standardized, and redundant SEC filings. The proposed 'corpus-side conditioning' principle is a coherent and potentially useful design lens, and the system-level results are broad: five datasets, multiple backbone models, matched retrieval budgets, human validation on 150 answers, and a long-context baseline. The strengths are the breadth of evaluation, the matched-budget study (Appendix B), the blind human check (Table 4), and the honest Limitations section, which explicitly identifies the missing separated baselines. If the mechanism-level claims are confirmed with direct ablations and the evaluation is tightened statistically, the work would be a solid systems contribution. As it stands, the core attribution of gains to the two named mechanisms is not yet isolated, and the gate's training labels derive from the very pipeline it is supposed to regulate, so the headline claims require further support.","major_comments":[{"comment":"The feature-gate training labels are generated by FinSAgent's own pipeline: an LLM synthesizes questions from random chunks, the full retrieval/reranking pipeline produces top-k candidates, and chunks matching the seed chunk are labeled positive. This is an in-distribution, pipeline-specific labeling scheme, not an external gold standard. The gate may therefore learn to recognize artifacts of the synthetic-question style or retrieval behavior rather than true evidential validity. This is load-bearing for the back-end claim (e.g., Table 3 Lotus recall 41→50 and Table 5 noise-sensitivity attribution). Please add a validation set with human/expert-annotated evidential-validity labels (or at least an independently sourced label set) and report gate performance on it, or otherwise show that the self-labeled training data does not drive the reported gains.","section":"§3.4.2, Appendix C.2"},{"comment":"Table 3 is a cumulative ablation in a fixed order; it does not include either mechanism alone or leave-one-out configurations. The authors explicitly concede in the Limitations paragraph: 'Our evidence for the two mechanisms currently rests on full-system ablations and fixed-pool comparisons.' As a result, the gains attributed to database-aware decomposition and feature-gated reranking could in principle come from generic added context, extra retrieval paths, or other confounds. The FinanceBench row is instructive: the full system drops relative to '+ DB-Aware Decomp.' (76 vs 80), which the text explains as an order/interference effect; this further shows cumulative contributions do not isolate stable mechanism-level effects. The planned separated baselines (static corpus-overview prompt vs. database-aware decomposition; NLI/claim verifier vs. feature-gated reranker) are exactly what is","section":"§4.3, Limitations (end of main text)"},{"comment":"The abstract claims 'a three-arm randomized online experiment with 1,000 anonymous user ratings,' but the body never describes such an experiment and §1 instead refers to '~1,400 anonymous user ratings.' The discrepancy is not merely numerical: no protocol, randomization details, rating instrument, or statistical analysis for this experiment is provided anywhere in the paper. Since the abstract's headline claim rests on this experiment, and the offline head-to-head results in Table 2 are reported without error bars, confidence intervals, or significance tests, the evaluation is not yet statistically grounded. Please unify the numbers, present the online-experiment protocol and results, and add variance estimates/significance tests (or clearly label the offline comparisons as descriptive).","section":"§1, §2, Abstract; §4.2"},{"comment":"The matched-budget study (Appendix B; Table 7) is a valuable control, but it gives the baselines FinSAgent's budgets with the two mechanisms simply disabled. This does not directly test the mechanisms: the baselines also lack the role-specific prompts, the orchestrator, and other pipeline components. Consequently the phrase 'the advantage comes from database-aware decomposition and feature-gated reranking, not from having more retrieval perspectives' (Appendix B) overstates what the design can establish. Please add a budget-matched baseline that includes a static corpus-overview prompt (or an equivalent non-database-aware decomposition) with the same role structure, and a baseline that replaces the feature gate with an NLI/claim verifier, so the two mechanisms are isolated.","section":"§4.3, Table 3; §4.2, Table 2"}],"minor_comments":[{"comment":"The LC (long-context) rows are shown as zeros on Lotus, SECQUE, FinDER, and Zeekr; it would be clearer to mark these as 'not applicable' or explain why long-context is infeasible there.","section":"Table 2"},{"comment":"FinGPT is described as a 'single-agent tool-augmented reasoner' but is given web/MCP access (§A.3). The comparison is reasonable, but the caption should note that FinGPT is not strictly a document-retrieval-only baseline.","section":"§4.2, Table 2"},{"comment":"RAGAS is mentioned without a reference; please add a citation for the RAGAS framework and for the noise-sensitivity metric.","section":"§4.5"},{"comment":"The abstract mentions 'five offline financial QA benchmarks' but the SECQUE subset is only 100 of 565 questions; please state this sampling explicitly in the abstract or keep the 'five benchmarks' phrasing but note the subset size in the main text (already in §4.1 and Appendix A.2).","section":"Abstract and §4.1"},{"comment":"The co-retrieval heatmap is useful, but the caption and text could clarify whether the counts are per-query averaged or raw totals across the full evaluated set.","section":"§4.3, Figure 3"},{"comment":"Some sentences are overly programmatic (e.g., 'we read this as evidence that the remaining bottleneck has moved downstream' in §4.8). A more measured phrasing would better match the results presented.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems submission with an honest limitations section, but the central mechanism-attribution claim is not yet supported by isolated ablations, and the abstract/body inconsistency around the online experiment needs to be resolved. I would not reject: the external benchmarks, matched-budget control, human validation, and long-context comparison are substantial. However, the authors themselves acknowledge the missing separated baselines, so a major revision that adds those experiments plus significance testing would be the appropriate path. No concerns about research integrity beyond the reporting inconsistency noted above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this as a solid systems paper with a hypothesis-testing gap. The authors frame SEC filing QA as \"prior–corpus misalignment\" and build FinSAgent: role-specialized agents, database-aware query decomposition, and a feature-gated reranker. The engineering is real, and the empirical base is broader than most papers in this space: five benchmarks, a matched-budget fairness study, a blind human check on SECQUE, RAGAS noise sensitivity, error analysis, and latency/token measurements. They also claim the gains hold across three different LLM backbones, which is a nice check against backbone confounds.\n\nThe central conceptual contribution—conditioning sub-queries on a lightweight corpus summary and using non-semantic features (retrieval provenance, lexical overlap, page number) to down-weight confident false positives—is worth taking seriously. The case study on boilerplate risk factors illustrates the intended mechanism concretely.\n\nThe soft spots are real and mostly about attribution. Table 3 is a cumulative ablation: single-agent, multi-agent, +feature-gate, +DB-aware, full. It never runs either mechanism alone or leave-one-out. The authors concede this in the Limitations section, saying their evidence \"rests on full-system ablations and fixed-pool comparisons\" and only plan separated baselines later. The FinanceBench dip (80 to 76 when adding DB-aware decomposition) also suggests order/interference effects. So the paper shows the whole system beats the baselines, but not that the two named mechanisms are the active ingredients.\n\nThe second concern is the feature gate's training data. Labels come from the system itself: an LLM writes questions from chunks, the pipeline retrieves candidates, and chunks matching the seed chunk are positive. That makes the gate a fit to the pipeline's synthetic-question distribution, not necessarily an independent measure of evidential validity. It's plausible the gate learns something about boilerplate, but the current evidence doesn't exclude the simpler story that it learns which chunks the retriever likes. This is an addressable issue, but it needs an external annotation source.\n\nAlso: the abstract promises a \"three-arm randomized online experiment with 1,000 anonymous user ratings,\" but the body never describes that experiment. The results section mentions ~1,400 anonymous ratings in passing, also without protocol. That discrepancy should be fixed before publication—either report the experiment properly or drop the claim.\n\nMissing error bars or significance tests on the headline comparisons is minor relative to the above, but still worth noting. And no code or data is released, which makes it hard to check the matched-budget claims.\n\nOverall, the paper deserves peer review: the task is important, the system is plausible, and the evaluation is far more thorough than average. But it needs major revisions, especially the mechanism isolation and the gating-label story. I'd send it to reviewers rather than desk reject, with the expectation that the authors be pushed on the abstract's online-experiment claim and on adding separated baselines. The core system result is probably real; the mechanism explanation is not yet proven.","headline":"A well-built SEC-filing QA system with a plausible but not yet isolated mechanism claim, and an abstract that overstates an online experiment the body never describes.","tokens_in":25139,"tokens_out":1543,"would_cite":false,"duration_ms":342412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinSAgent argues that SEC filing QA fails because model priors misalign with how filings are written, and corrects both query generation and evidence selection with corpus-side conditioning.","keywords":["SEC filings","retrieval-augmented generation","multi-agent systems","question answering","evidence grounding","query decomposition","reranking","prior–corpus misalignment"],"falsifier":"Train the feature gate on human-annotated evidence-validity labels, or replace it with an independent claim-entailment verifier, holding everything else fixed. If retrieval coverage and answer correctness do not improve over the ungated reranker, the central mechanism is an artifact of self-labeling; if the gains persist, the mechanism genuinely separates validity from semantics.","tokens_in":24186,"feed_emoji":"📊","tokens_out":3453,"duration_ms":39134,"temperature":0.7,"pith_summary":"This paper argues that question answering over SEC filings fails for a specific reason: the language models that generate search queries and rank retrieved passages rely on generic knowledge of what a financial answer should look like, while valid evidence is written and organized according to the filings' own conventions. It calls this mismatch prior–corpus misalignment and claims it corrupts both ends of retrieval: query generation misses corpus-specific evidence, and semantic reranking promotes boilerplate that resembles the question but is not valid evidence for that company. The proposed system, FinSAgent, injects corpus-side conditioning at both ends — role-specialized agents tied to the 10-K item structure, sub-queries generated with a lightweight view of the local filing database, and a learned feature gate that demotes confident false positives. Across five benchmarks and a randomized user study, the system reports better retrieval coverage and answer correctness than strong baselines, and it beats full-document long-context prompting on the benchmark most favorable to that alternative.","feed_headline":"SEC filing QA is a retrieval-planning problem, not a long-context problem","feed_subtitle":"Corpus-side conditioning at both query and rerank stages lifts answer correctness over baselines on five benchmarks.","key_machinery":"The load-bearing mechanism is the feature-gated reranker: a semantic cross-encoder score is multiplied by a penalty (1 − λ(1 − p̂)), where p̂ is a gradient-boosted tree's estimate of evidential validity from 31 non-semantic features such as path provenance, path scores, lexical overlap, and chunk metadata. Because the penalty scales with the reranker's own confidence, high-scoring boilerplate is demoted more than low-scoring chunks. The second mechanism is database-aware query decomposition: before generating sub-queries, each agent retrieves a small set of section-level summaries of the local corpus and conditions its sub-queries on that view. Together they instantiate the paper's principle","core_discovery":"The paper's central discovery is that the two observed failure modes in SEC filing QA — missed evidence and false-positive boilerplate — have one cause: prior–corpus misalignment. It claims that conditioning retrieval planning on a lightweight view of the local corpus and gating reranking by non-semantic validity features corrects both ends. The evidence includes component ablations showing each addition improves recall on at least some corpora, matched-budget comparisons showing the gains are not from extra retrieval perspectives, and a double-dissociation showing the two mechanisms help different corpora for different reasons. The authors state the conclusion directly: SEC filing QA is bes","pith_inferences":["Editorial inference: the prior–corpus misalignment framing suggests a direct transfer test — applying the same database-aware decomposition and feature gate to another standardized corpus should reproduce the recall and correctness gains if the cause is truly structural rather than SEC-specific.","Editorial inference: the paper's own future-work comparison against an NLI-style claim verifier would clarify whether the statistical gate learns evidential validity or merely the style of the synthetic training questions; a human-labeled validation set would settle the same question.","Editorial inference: the online user study and the five benchmarks measure different question mixes; breaking down gains by question type (quantitative, legal, market, company) could show which role specializations earn their inference cost and which are optional."],"forward_implications":["If correct, the same two-end correction should transfer to any standardized, redundancy-heavy corpus, such as patents, clinical trial reports, or regulatory disclosures, where boilerplate can outrank company- or case-specific evidence.","Reranking pipelines that rely purely on semantic similarity may be leaving cheap, non-semantic signals on the table; a learned gate over retrieval-path and lexical features can separate evidential validity from topical similarity.","Planned, corpus-aware retrieval can outperform handing an entire document to a long-context model, even on a single-document benchmark designed to favor the long-context approach.","The error analysis implies that once retrieval and comprehension failures are suppressed, the remaining bottleneck shifts to fine-grained numerical precision, pointing to a clear next target for financial QA systems."],"fun_headline_variants":["Fix SEC filing QA by aligning model priors with corpus structure","SEC filing QA errors traced to prior–corpus misalignment","FinSAgent: corpus-aware query and rerank for SEC filing QA","One root cause: model priors clash with filing corpus structure"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The gate's training labels come from the system's own pipeline — an LLM writes synthetic questions from random chunks, the retrieval pipeline returns candidates, and chunks matching the seed chunk are called positive — so the gate may be learning to recognize the synthetic question style rather than true evidential validity.","fun_headline_variants_meta":{"raw":{"variants":["Fix SEC filing QA by aligning model priors with corpus structure","SEC filing QA errors traced to prior–corpus misalignment","FinSAgent: corpus-aware query and rerank for SEC filing QA","One root cause: model priors clash with filing corpus structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3528,"prompt_tokens":785,"completion_tokens":2743,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2679}},"tokens_in":529,"tokens_out":2743,"duration_ms":22572,"temperature":1.0,"reasoning_tokens":2679,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:59:04.503798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the feature gate on human-annotated evidence-validity labels, or replace it with an independent claim-entailment verifier, holding everything else fixed. If retrieval coverage and answer correctness do not improve over the ungated reranker, the central mechanism is an artifact of self-labeling; if the gains persist, the mechanism genuinely separates validity from semantics.","supporting_citations":[],"review_version":1}