{"id":"50af195e-ab5b-43c8-8215-ed938fb6108a","arxiv_id":"2504.14493","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"FinSage reports a 92.51% retrieval recall and a 49.66% financial QA accuracy, but the main private-dataset metrics are in-sample and no code or data is released.","lead":"FinSage is an AI question-answering system for financial filings that combines multi-modal document preprocessing, multiple retrieval strategies, and a specialized re-ranker. The paper reports large accuracy gains on FinanceBench and a custom Lotus Technology dataset, but the evaluation has in-sample training, missing artifacts, and inconsistent tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Company-dataset headline numbers are likely in-sample: HyDE is fine-tuned on the custom Company dataset (Sec. 3.2), the DPO reranker is trained on single-company filings and iteratively adjusted until it performs on a 'new test set' (Sec.","rationale":"The reader's weakest assumption is the right one, and the manuscript text makes it more than a missing-split worry. Section 3.2 explicitly ties HyDE instruction fine-tuning to the custom Company dataset; Section 4.2.2 ties the DPO reranker to single-company financial filings; and Section 3.3 describes retraining against a test set until performance is satisfactory. Together these make the Company-dataset results in-sample unless a disjoint split is demonstrated. This matters because the 92.51% recall and the reranker comparisons in Figure 5 are the primary evidence that the multi-path retrieval and DPO reranking components help. The FinanceBench comparison is also uncontrolled: FinSage is run through its own FFP pipeline and GPT-4o evaluator while baseline numbers are quoted from original papers, so the 24.06-point gain is not a controlled apples-to-apples measurement. The deployment evidence (2,702 queries, 4.19/5 satisfaction) is real user feedback but does not validate the research claims, since there is no held-out experimental control. A concrete release-and-recompute check would settle the matter. I agree with the reader's REJECT assessment; the burden of proof is on the authors to show the evaluation set was not touched by training or iterative tuning, and currently the text suggests the opposite.","tokens_in":22738,"tokens_out":7756,"duration_ms":64967,"concrete_test":"Ask the authors to release (i) the HyDE SFT training pairs and the DPO reranker preference data (queries, positives, negatives) and (ii) the exact 75 Company queries with their annotated chunk IDs. Check exact and near-duplicate overlap between training queries and the 75 test queries, and between any training chunk and the annotated ground-truth chunks. Then re-run Table 1 and Table 2 Company rows on a disjoint held-out split (e.g., leave-one-filing-out or a chronological split) without the Section 3.3 iterative retuning loop. If recall or QA accuracy drops materially (e.g., more than 5 points), the reported gains are in-sample. Also rerun the FinanceBench comparison with all baselines implemented in the same pipeline and the same LLM judge before claiming a 24.06-point gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FinSage's central numerical claims rest on the Company dataset being an independent evaluation set. The paper never states such a split, and several sections imply the opposite. Section 3.2 says the HyDE model is given domain knowledge through instruction fine-tuning on 'our custom Company dataset' before retrieval is measured on that same Company dataset (Table 1, Figure 5). Section 4.2.2 says the DPO reranker is trained on single-company financial filings; Section 3.3 then describes an iterative loop in which, 'when the model fails to perform on a new test set, the retrieved documents are re-annotated... and retrained... repeated until a satisfactory performance on the new test set is attained.' If the 75 expert questions or their annotated chunks participate in either training or in that iterative tuning loop, the reported 92.51% recall, the reranker gains, and the 85.33%/88.00% QA scores are measures of fit to the evaluation set, not of system quality. The FinanceBench results are not a substitute: Table 2 compares FinSage's own GPT-4o-judged run against numbers taken from the original papers, so the '24.06 percentage point gain' mixes evaluation protocols and is not controlled. No code or data artifacts are released, so the overlap cannot currently be checked by a reader.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinSage, a RAG system for question answering over financial filings. FinSage consists of three components: a multi-modal preprocessing pipeline (FFP) that unifies text, tables, and figures into text chunks with chunk-level metadata; a multi-path retrieval pipeline (MPR) that combines BM25, dense retrieval (BGE-M3), metadata-based retrieval, and HyDE query expansion; and a domain-specialized reranker trained with a claimed Direct Preference Optimization (DPO) objective. The authors report 92.51% retrieval recall on a private 75-question Company dataset built from Lotus Technology filings, 49.66% LLM-judged accuracy on FinanceBench (a claimed 24.06 percentage point improvement over cited baselines), and 85.33%/88.00% QA accuracy on the Company dataset. The system is said to be deployed and to have served over 1,200 users.","tokens_in":23075,"tokens_out":6087,"duration_ms":51285,"significance":"A well-engineered, deployed financial QA system is a worthwhile application contribution, and the architecture sensibly integrates known techniques (metadata-aware retrieval, HyDE, multi-path fusion, and reranking with a time bonus). If the evaluation were properly controlled, the 92.51% recall result on a private dataset would demonstrate value for a specific deployment. However, as presented, the core experimental claims are not supported: the headline Company-dataset numbers appear to be in-sample, the FinanceBench comparison mixes evaluation protocols and judges, and the reranker is not actually trained with DPO as formulated. The paper also provides no code or data release, so the central findings cannot be independently checked. The extensive deployment analytics in Appendix E are helpful but do not substitute for a controlled research evaluation.","major_comments":[{"comment":"The central evaluation on the Company dataset is not shown to be held-out. Section 3.2 states that HyDE is instruction-fine-tuned \"on our custom Company dataset,\" Section 4.2.2 states that the Document Re-ranker is \"trained on single-company financial filings,\" and Section 3.3 describes an iterative loop in which retrieved documents are re-annotated and the model is retrained \"until a satisfactory performance on the new test set is attained.\" Since Table 1, Figure 5, and the Company rows of Table 2 are all evaluated on the manually annotated Company dataset described in Section 4.1, the 92.51% recall and the reranker gains are at risk of being in-sample measurements of fit to the evaluation set. The paper must state explicitly how the 75 expert questions and their annotated chunks are disjoint from all training and tuning data, provide the actual split, and ideally test on a different company's filings to establish generalization.","section":"Sec. 3.2, 3.3, 4.2.1, 4.2.2, 4.3; Table 1, Figure 5, Table 2"},{"comment":"The FinanceBench comparison is not controlled. The table footnote says results with \"*\" are from the authors' experiments and all others are \"from the original papers,\" so FinSage's GPT-4o-judged LLM accuracy is compared with numbers obtained under different retrieval settings, chunk sizes, generation prompts, and possibly different evaluation metrics. The claimed 24.06 percentage point improvement over the \"best baseline\" (49.66% vs. 25.60%) is therefore not established. The authors should re-run all baselines in their own pipeline with the same judge, the same protocol, and the same retrieval and generation settings, and report variance or significance tests.","section":"Section 5.3, Table 2"},{"comment":"The loss in Eq. (2) is not Direct Preference Optimization. DPO, as introduced by Rafailov et al., optimizes a language model policy against a reference model with a KL penalty; Eq. (2) is a pairwise logistic (contrastive) loss on reranker scores. The paper's third contribution (\"fine-tuned via Direct Preference Optimization\") is therefore not supported by the formulation, and the cited reference [49] is an instruction-tuning survey rather than the DPO paper. The authors should either implement and report actual DPO or relabel the objective as a pairwise ranking loss and adjust the contribution claim accordingly.","section":"Section 3.3, Eq. (2)"}],"minor_comments":[{"comment":"The abstract contains an incomplete sentence: \"FinSage achieves an impressive recall of 92.51% on 75 expert-curated questions derived from surpasses the best baseline method...\" which appears to be a copy-paste artifact and should be fixed.","section":"Abstract"},{"comment":"The main-text Figure 5 reports values that correspond to the R=10 configuration in Table 7 but the caption and Section 5.2 do not state the candidate-bundle size R; the text \"from 78.8% to 38.8%\" for precision also matches R=10 values only. Please make the configuration explicit and consistent between the figure, table, and text.","section":"Figure 5, Table 7, Section 5.2"},{"comment":"The time bonus function f(\\tilde{m}_i) and the scale \\beta are never defined; the paper only gives a verbal description of a one-year cutoff. A concrete formula (e.g., a decay function of publication date) is needed for reproducibility.","section":"Eq. (1), Section 3.3"},{"comment":"The definition of \"Normalized Recall\" (capping the denominator at min(ground truth chunks, K)) makes it an upper bound of standard recall and not comparable to traditional recall values; Table 1 reports \"Recall\" without specifying whether the capped or uncapped variant is used.","section":"Section 4.3"},{"comment":"The GraphRAG/LightRAG comparison uses different numbers of questions (71 vs. 75) and a \"Faithful Evaluation Score\" whose definition and rating scale are not given in the main text; please state the evaluation protocol and score rubric explicitly.","section":"Tables 3 and 4, Appendix F"},{"comment":"Typographical errors in the command parameters: \"gpu-memoty-utilization\" should be \"gpu-memory-utilization\" and \"engorce-eager\" should be \"enforce-eager\".","section":"Appendix C.1, Table 6"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to describe an industry deployment, and the evaluation design seems oriented toward showcasing the deployed system rather than establishing a scientific claim. The potential train/test leakage on the Company dataset and the uncontrolled FinanceBench comparison are load-bearing problems that would require a fundamentally new experimental design (proper held-out evaluation, re-run baselines under the same protocol, and a correct formulation of the reranker objective). These are not local fixes, so I recommend rejection. If the authors redo the evaluation with a genuine held-out set and controlled baselines, a resubmission could be considered. The deployment analytics in Appendix E, while interesting, do not compensate for the lack of a controlled study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: FinSage is a coherent, well-engineered RAG stack for SEC filings—multi-modal preprocessing with metadata summaries, four-path retrieval (BM25, BGE-M3 dense, metadata, HyDE), chunk bundling, and a DPO-tuned reranker with a time bonus. The pieces are known, but the specific integration is new and the deployment stats (2,702 queries, 4.19/5 satisfaction) suggest real industrial use. I believe the architecture works as described.\n\nThe problem is the evaluation. The headline 92.51% recall on the 75-question Company dataset is very likely in-sample. Section 3.2 says HyDE is instruction fine-tuned on \"our custom Company dataset,\" and Section 4.2.2 says the DPO reranker is trained on single-company financial filings. Section 3.3 then describes an iterative loop: when the model fails on a \"new test set,\" the retrieved documents are re-annotated and the model retrained, \"repeated until a satisfactory performance on the new test set is attained.\" No split between training and the 75 expert questions is ever stated. That is training to the evaluation set. The 85.33%/88.00% QA scores and the reranker gains in Figure 5 inherit the same problem.\n\nThe FinanceBench comparison is also uncontrolled. Table 2 puts FinSage's own GPT-4o-judged run (49.66% LLM, 57.05% manual) next to numbers from the original papers, which used different judges, prompts, and chunking. The \"24.06 percentage point gain\" is not an apples-to-apples comparison. The baselines are not re-run under FinSage's setup except for a few asterisked entries, and no code or data is released to verify overlap.\n\nMinor issues: the abstract has a malformed sentence, the appendix describes deduplication with TF-IDF while the main text uses dense cosine similarity, and there are typos. These are cosmetic.\n\nWhat's genuinely useful: the paper is a reasonable blueprint for a production financial QA system. The component choices are defensible, and the deployment evidence, while anecdotal, is more than most systems papers provide. If the authors release the Company dataset, state the train/test split, and re-run FinanceBench baselines under a single protocol, this becomes a solid systems paper.\n\nMy recommendation: send it to peer review—the engineering is worth referee time—but expect the evaluation to be the focus. As submitted, the central claims are not supported; I'd vote reject pending the re-analysis.","headline":"A coherent industrial RAG stack whose headline numbers are undercut by in-sample evaluation on the private dataset and an uncontrolled FinanceBench comparison; worth a referee, but rejection is right for the current evidence.","tokens_in":23621,"tokens_out":2571,"would_cite":false,"duration_ms":21432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinSage claims that a multi-path retrieval pipeline with a DPO-tuned reranker raises financial-filing question answering by 24 percentage points over existing baselines.","keywords":["Retrieval-Augmented Generation","Financial filings","Question answering","Multi-path retrieval","HyDE query expansion","Direct preference optimization","Document preprocessing","FinanceBench"],"falsifier":"Retrain the HyDE generator and the reranker from scratch using only filings and questions that are temporally and company-wise separated from the 75 annotated evaluation questions, then rerun retrieval and report recall. If 92.51% recall drops materially, the headline number depends on train/test overlap rather than on the architecture.","tokens_in":22591,"feed_emoji":"📈","tokens_out":5717,"duration_ms":48606,"temperature":0.7,"pith_summary":"This paper is trying to establish that a retrieval-augmented generation system built specifically for financial filings can answer compliance questions more accurately than generic RAG pipelines. The proposed system, FinSage, combines multimodal document preprocessing, four retrieval paths, and a reranker fine-tuned with direct preference optimization. On a private dataset of 75 expert-curated questions from a single company's filings, it reports 92.51% retrieval recall; on the public FinanceBench benchmark, it reports 49.66% LLM-judged accuracy, a 24.06 percentage point gain over the strongest baseline it compares with. The authors argue this shows domain-specific preprocessing and retrieval design matter more than simply scaling the language model.","feed_headline":"RAG system lifts financial filing answers by 24 points","feed_subtitle":"FinSage routes queries through four retrieval paths and a preference-tuned reranker for compliance-heavy filings.","key_machinery":"The load-bearing mechanism is the multi-path retrieval plus preference-tuned reranking loop. BM25 supplies lexical matching, BGE-M3 supplies dense semantic matching, the metadata retriever matches the query against each chunk's heading-plus-section-summary so that retrieving one metadata instance pulls in all chunks of that section, and HyDE generates hypothetical answers to close lexical and semantic gaps. The chunk bundling module then expands each retrieved chunk to adjacent chunks with cosine similarity above 0.85, and the document reranker combines a cross-encoder score with a metadata recency bonus under a DPO-trained objective, so compliance-relevant and current chunks are preferred.","core_discovery":"The central claim is that financial-filing QA improves when preprocessing, retrieval, and reranking are each adapted to the domain instead of using off-the-shelf RAG. FinSage converts tables and figures into textual narratives, deduplicates and resolves co-references, attaches section summaries as metadata, and then retrieves through BM25, a dense retriever, a metadata-aware retriever, and a HyDE retriever whose small generator has been fine-tuned on financial data. Retrieved chunks are expanded into bundles of neighboring chunks, reranked by a cross-encoder that was preference-tuned on single-company filings and that adds a recency bonus, and then fed to an LLM for answer generation. The paper reports 92.51% recall on 75 expert-annotated company questions and a 24.06 percentage point accuracy gain over the best baseline on FinanceBench.","pith_inferences":["Beyond the paper: the reported component gains are not cleanly attributed to FFP, MPR, or DRR; an ablation that holds each component fixed while toggling the others would quantify each one's contribution.","Beyond the paper: the reranker's time bonus implies recency is load-bearing; a testable extension is to ask the same factual question about an old and a new filing with identical content and check whether ranking consistently favors the newer one.","Beyond the paper: because the 75-question evaluation is drawn from one company's filings, cross-company or portfolio-level questions are an open test; a mixed-company vector store would reveal whether the metadata and HyDE paths transfer.","Beyond the paper: metadata retrieval returns entire semantic sections when one metadata instance matches, which likely explains the recall gain but may also inflate false positives; measuring precision per semantic segment could expose where the approach overshoots."],"forward_implications":["If FinSage's results hold, combining several retrieval paths with query expansion recovers relevant sections that dense and lexical retrieval alone miss, without needing a larger LLM.","A fine-tuned 7-billion-parameter HyDE generator can outperform a 72-billion-parameter counterpart on this retrieval task, suggesting domain data can substitute for model scale.","A preference-tuned reranker yields most of its precision and recall gains in the top-5 to top-10 range, so production systems can keep retrieved context small.","On this benchmark, graph-based RAG solutions do not beat a well-preprocessed multi-path vector pipeline, so compliance QA deployments do not have to adopt graph representations to get faithful answers.","Reported end-to-end accuracy of 49.66% (LLM-judged) and 57.05% (manual) on FinanceBench, and 85.33% and 88.00% on the company dataset, indicate the pipeline's gains carry through to final answers, not just retrieval."],"supporting_citations":[{"why":"Supplies the FinanceBench dataset and the shared-vector-store baseline that FinSage must beat.","marker":"[15]"},{"why":"Provides the financial-report chunking baseline Base256 and motivates the table and figure textualization approach.","marker":"[17]"},{"why":"Provides the reranker-based financial RAG baseline that most closely matches FinSage's configuration.","marker":"[36]"},{"why":"Supplies the BGE-M3 embedding model used for dense chunk and metadata embeddings.","marker":"[10]"},{"why":"Supplies the HyDE query-expansion technique that the HyDE retriever is built on.","marker":"[48]"},{"why":"Supplies the InPars-style dataset generation method used to fine-tune the HyDE generator on financial data.","marker":"[7]"},{"why":"Supplies the MinerU PDF parsing tool used in the FFP textual encoding step.","marker":"[12]"},{"why":"Supplies evidence that top-5 to top-10 chunks are sufficient for effective LLM answers, used to set K.","marker":"[24]"}],"fun_headline_variants":["FinSage: multi-path RAG beats financial filing baselines by 24%","Domain-tuned RAG scores 92.5% recall on financial filings","Four retrieval paths plus preference reranking lift finance QA","FinSage RAG: compliance-aware retrieval improves answers 24%","Financial filing QA: RAG with HyDE and metadata retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 75 expert questions and their annotated chunks must have been held out from training and tuning of the HyDE generator and the DPO reranker, but the paper never states that split, and both components are described as trained on finance data from the same company.","fun_headline_variants_meta":{"raw":{"variants":["FinSage: multi-path RAG beats financial filing baselines by 24%","Domain-tuned RAG scores 92.5% recall on financial filings","Four retrieval paths plus preference reranking lift finance QA","FinSage RAG: compliance-aware retrieval improves answers 24%","Financial filing QA: RAG with HyDE and metadata retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1515,"prompt_tokens":1000,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":616,"tokens_out":515,"duration_ms":4624,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:47:20.878607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the HyDE generator and the reranker from scratch using only filings and questions that are temporally and company-wise separated from the 75 annotated evaluation questions, then rerun retrieval and report recall. If 92.51% recall drops materially, the headline number depends on train/test overlap rather than on the architecture.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the financial-report chunking baseline Base256 and motivates the table and figure textualization approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MinerU PDF parsing tool used in the FFP textual encoding step."}],"review_version":1}