{"id":"23704f1e-4919-4891-b40d-b4834059f7a6","arxiv_id":"2506.16037","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A standard RAG pipeline with an undefined multi-hop module is reported to outperform baselines on financial QA datasets, without code or data.","lead":"This paper proposes FinLLaMA-RAG, a retrieval-augmented question answering system that combines LLaMA 3 embeddings with cosine retrieval, attention-weighted context fusion, and an unspecified iterative reasoning step. It reports higher scores than baselines on financial QA benchmarks, but provides no code, data, or reproducible evaluation details.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The multi-hop reasoning module LLaMA3hop in Eq. (5) is never specified, so the central claim that FinLLaMA-RAG outperforms RAG baselines is unsupported; an identity ablation would test whether the reported gains have any mechanism.","rationale":"The reader's weakest_assumption correctly identifies the undefined multi-hop module as the load-bearing point. I agree. For the central claim to hold, the multi-hop reasoning must be an implementable, trainable component that adds value beyond ordinary RAG; Eq. (5) gives no such component. The issue is internal, not just a disagreement with the community: the equations that define the method are incomplete, and Eq. (8) uses T inconsistently, making the training objective ill-posed as written. The evaluation section reinforces the concern: Table II's ablation omits the baselines it claims to compare, and none of the reported numbers come with error bars or reproducibility artifacts. I found no machine-checked proofs, no released code, and no parameter-free derivation that would independently support the headline result. I am not claiming the authors fabricated the numbers; the absence of specification is sufficient to make the central claim unsupported. Thus the REJECT verdict stands, and I would not change the reader's verdict.","tokens_in":4825,"tokens_out":5034,"duration_ms":55599,"concrete_test":"Re-run the full pipeline with Eq. (5)'s LLaMA3hop replaced by the identity function D_hop^(t)=D_agg for all t, keeping retrieval, fusion, generation, Eq. (9)'s loss, and the Table I evaluation protocol identical. If the identity version reproduces the reported FinLLaMA-RAG row (nDCG@10 0.62, BLEU 30.5, ROUGE-L 35.2, F1 0.75/0.78), the multi-hop module contributes nothing measurable and the central claim reduces to standard RAG. If the authors cannot supply the missing LLaMA3hop definition and training details needed to run this ablation, the empirical claim is not independently checkable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FinLLaMA-RAG outperforms existing retrieval-augmented and generative baselines in document-level and financial QA. That claim rests on the multi-hop reasoning module. Eq. (5) defines D_hop^(t)=LLaMA3hop(D_hop^(t-1),q), but LLaMA3hop is never defined: no architecture, parameter count, input/output dimensionality, attention pattern, number of iterations, or training procedure is given. The figure described as its pipeline is a generic diagram with no equations. This is not a minor omission: D_hop^(T) is the only input to the generation module in Eq. (6), so every reported improvement is attributed to an object the paper does not specify. The internal inconsistency of the loss makes the problem worse. Eq. (8) writes L_generation = -sum_{t=1}^{T} log P(r_t | r_<t, D_hop^(T), q), using T both for the number of hops in Eq. (5) and for the token position of the answer r_t. As written, the objective is not a well-defined function of the model's outputs. Table II's ablation rows for Retrieval-Only and Generation-Only models do not include the baselines named in Table I, and no error bars, significance tests, dataset splits, or code are provided. If LLaMA3hop were replaced by the identity function, the framework would reduce to cosine retrieval, softmax-weighted context fusion, and LLaMA 3 generation, essentially standard RAG. The paper gives no evidence that the reported numbers would change.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinLLaMA-RAG, a retrieval-augmented generation framework built on LLaMA 3, combining dense retrieval, attention-based context fusion, a multi-hop reasoning module, and a joint retrieval-plus-generation loss. It reports nDCG@10, BLEU, ROUGE-L, and F1 results on five financial QA datasets and claims to outperform existing retrieval-augmented and generative baselines. The paper also discusses applications in tax compliance and fraud investigation.","tokens_in":5208,"tokens_out":2817,"duration_ms":35074,"significance":"If the framework were fully specified and the empirical claims were supported by reproducible experiments, this would be a useful contribution to document-level and financial question answering, an area where multi-hop reasoning over long documents is genuinely challenging. However, the central multi-hop reasoning module is never defined, the loss function is internally inconsistent, and the experimental section lacks the data, hyperparameters, error bars, and ablations needed to verify the headline claim. As presented, the manuscript provides little beyond a high-level RAG sketch with unreviewable numbers, so its significance is limited.","major_comments":[{"comment":"The central module LLaMA3hop is never defined. The recurrence D_hop^(t) = LLaMA3hop(D_hop^(t-1), q) is the only mechanism claimed to enable multi-hop reasoning, but the paper gives no architecture, parameterization, input or output dimensionality, training procedure, or iteration count for this function. Since D_hop^(T) is the sole context passed to the generation module in Eq. (6), every reported improvement is attributed to an unspecified object. If LLaMA3hop were replaced by the identity function, the framework would reduce to cosine retrieval, softmax-weighted context averaging, and standard LLaMA 3 generation; the paper provides no evidence that any observed gains would survive that substitution.","section":"Section III-D, Eq. (5)"},{"comment":"The generation loss is not a well-defined objective as written. The symbol T is used both as the number of multi-hop iterations in Eq. (5) and as the sequence length of the answer tokens r_t in Eq. (8), while D_hop^(T) is treated as a fixed context for all token positions. The expression L_generation = -sum_{t=1}^{T} log P(r_t | r_<t, D_hop^(T), q) therefore cannot be evaluated without additional, unspecified conventions about how the hop count and token length are reconciled. This undermines the paper's claim to a joint optimization strategy.","section":"Section III-F, Eq. (8)"},{"comment":"The empirical evidence is not auditable. Table I reports metrics for five datasets, but Table II presents only a single set of aggregate numbers and omits all of the named baselines from Table I. No standard deviations, confidence intervals, significance tests, dataset splits, or hyperparameters are provided, and no code or data are released. Under these conditions, the central claim that 'FinLLaMA-RAG outperforms existing retrieval-augmented and generative baselines' is not statistically or reproducibly supported.","section":"Section V, Tables I and II"},{"comment":"The ablation study cannot establish the contribution of the proposed modules. The rows 'Retrieval-Only Model' and 'Generation-Only Model' are not defined in the methodology, and it is unclear which components they retain and which they remove. More importantly, the ablation does not include the baselines from Table I, so it does not show whether the multi-hop module, the contextual fusion layer, or the joint loss is responsible for any observed difference. An identity ablation of Eq. (5) is needed to test whether the reported gains are due to LLaMA3hop at all.","section":"Section V, Table II"}],"minor_comments":[{"comment":"The keyword heading contains the typo 'Kerwords' instead of 'Keywords'.","section":"Abstract"},{"comment":"Several references in the related work are unrelated to retrieval-augmented generation or question answering, including [5] on 3D reconstruction, [6] on breast cancer prediction, and [11] on cloud scheduling; these do not support the surrounding claims about RAG or financial QA.","section":"Section II"},{"comment":"The notation is ambiguous: Eq. (2) writes d_i for both the retrieved chunk and its embedding, while the contextual fusion layer in Eq. (3) treats d_i as an embedding vector; the distinction should be made explicit.","section":"Section III-C and Eq. (2)"},{"comment":"The training loss curves and 'training indicators' plots lack axis labels, units, and a clear description of what is being shown, which makes them unverifiable and of limited use to readers.","section":"Figures 3 and 5"},{"comment":"The text mentions a 're-ranking mechanism' and Figure 4 compares initial retrieval scores with re-ranked scores, but no re-ranking method is defined anywhere in the paper.","section":"Section III-G"}],"recommendation":"reject","confidential_remarks":"For the editor: the manuscript's central mechanism is undefined, the loss function contains a notational collision that makes it unimplementable, and the experimental results are presented without the reproducibility information expected for a journal submission. The paper does not currently meet the technical bar for publication, and the missing specification of the core module would require rewriting the methodology rather than a local revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The reader's take is right. The central claim that FinLLaMA-RAG outperforms existing RAG baselines cannot be evaluated because the paper never defines LLaMA3hop, the module that supposedly does the multi-hop reasoning. Eq. (5) just names it; there is no architecture, no parameterization, no training procedure, no ablation varying hop count. Since D_hop^(T) is the only input to the generation module, every reported gain is attributed to a black box the paper does not specify. That is a load-bearing flaw, not a missing detail.\n\nWhat is actually new? Very little. The pipeline is dense query embedding, cosine retrieval, softmax attention over retrieved chunks, an iterative update that is notationally recursive, and LLaMA-3 generation. The joint loss in Eq. (9) is a weighted sum of two standard losses. The writing is clear for the standard parts, and the evaluation section at least names five financial QA datasets, but the ablations are incomplete: Table II's Retrieval-Only and Generation-Only rows do not include the baselines from Table I, so there is no way to see whether the multi-hop module adds anything over a simple retriever-plus-LLM. No error bars, significance tests, code, data, or hyperparameters are provided, so the numbers in Table I are just claims.\n\nThe loss definition has a real internal inconsistency: Eq. (8) uses T both as the number of hops and as the token index of the answer, making the generation objective not well-defined as written. The related work is padded with irrelevant papers (3D reconstruction, breast cancer prediction, cloud scheduling, privacy-preserving LLMs) that do not bear on the method. The broader applications to fraud detection and tax assistance are speculative and unsupported by any experiment. These are internal, structural defects, so the stress-test's suggestion is apt: replacing LLaMA3hop with the identity would reduce the framework to standard RAG, and the paper gives no evidence the reported numbers would change.\n\nThis is not a serious submission. A desk reject is appropriate. No part of the method is derivable or verifiable, and the core module is undefined. If the authors later specify the module, release code and data, and run proper ablations with error bars, a revised version might deserve a look, but this manuscript does not.","headline":"A standard RAG pipeline with an undefined multi-hop module; the central empirical claim is unsupported because the main component is never specified.","tokens_in":5721,"tokens_out":1462,"would_cite":false,"duration_ms":18802,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a LLaMA 3-based RAG system with an iterative multi-hop reasoning module outperforms prior retrieval-augmented and generative baselines on five financial question-answering benchmarks.","keywords":["retrieval-augmented generation","multi-hop reasoning","financial question answering","LLaMA 3","context fusion","document-level QA","joint loss","nDCG@10"],"falsifier":"Run the full framework and a $T=1$ variant with identical data, compute budget, and hyperparameters; if nDCG@10, BLEU, ROUGE-L, and F1 on FinDER and FinQA do not improve when $T>1$, the central claim that multi-hop reasoning drives the gains is refuted.","tokens_in":4640,"feed_emoji":"📄","tokens_out":7912,"duration_ms":70039,"temperature":0.7,"pith_summary":"This paper proposes FinLLaMA-RAG, a retrieval-augmented generation framework built on LLaMA 3 for document-level question answering. It claims that adding an iterative multi-hop reasoning module and a contextual fusion layer lets the model combine evidence scattered across document chunks, yielding answers that are more accurate and contextually grounded than standard RAG and generative baselines. The paper reports consistent gains on five financial QA datasets, FinDER, FinQABench, FinanceBench, TATQA, and FinQA, with FinLLaMA-RAG reaching 0.62 nDCG@10, 30.5 BLEU, 35.2 ROUGE-L, and F1 scores of 0.75 and 0.78. These numbers, if reproducible, would make the framework a practical recipe for answering questions that require reasoning across multiple documents, with applications in tax compliance, fraud investigation, and regulatory review.","feed_headline":"Multi-hop RAG with LLaMA 3 tops financial QA baselines","feed_subtitle":"FinLLaMA-RAG reports 0.62 nDCG@10 and 30.5 BLEU, beating GPT-3 and FinBERT on five benchmarks.","key_machinery":"The machinery is a recurrent update over an attention-weighted aggregate of retrieved chunk embeddings: $D_{\\text{hop}}^{(t)} = \\mathrm{LLaMA3hop}(D_{\\text{hop}}^{(t-1)}, q)$, $D_{\\text{hop}}^{(0)} = D_{\\text{agg}}$, where $D_{\\text{agg}} = \\sum_i \\alpha_i d_i$ and $\\alpha_i$ are cosine-similarity attention weights. Each iteration re-reads the aggregated context against the query, so the final representation carries evidence selected across multiple hops. The paper names the component LLaMA3hop and defines it by this iterative input-output behavior, then couples it to the generator through a joint loss $\\mathcal{L}_{\\text{total}} = \\lambda_{\\text{retrieval}} \\mathcal{L}_{\\text{retrieval}} + \\lambda_{\\text{generation}} \\mathcal{L}_{\\text{generation}}$ that ties retrieval precision to generation quality in one training signal.","core_discovery":"The central claim is that multi-hop reasoning over retrieved chunks is what lets the generator answer questions whose evidence is distributed across documents. The proposed update $D_{\\text{hop}}^{(t)} = \\mathrm{LLaMA3hop}(D_{\\text{hop}}^{(t-1)}, q)$ is applied $T$ times, starting from an attention-weighted aggregate $D_{\\text{agg}}$ of the top-$k$ retrieved chunks; the final state $D_{\\text{hop}}^{(T)}$ is then passed to the LLaMA 3 generator. Retrieval and generation are trained jointly through a weighted sum of a retrieval likelihood and a generation cross-entropy, so the two objectives improve together. Evaluated on five financial QA datasets, the framework reports higher nDCG@10, BLEU, ROUGE-L, and F1 than BERT-based retrieval, traditional RAG, FinBERT, and GPT-3.","pith_inferences":["The paper never specifies the internal architecture of LLaMA3hop; a natural extension is to test whether the iterative recurrence itself, rather than a particular reader network, is what drives the reported gains.","One testable extension is to apply the framework to legal or medical documents; if the gains are caused by multi-hop aggregation, similar margins should appear wherever evidence is scattered across sections.","Because the attention weights and per-hop states are inspectable, the framework could be adapted to produce an audit trail of which chunks influenced the final answer, which would matter in regulated settings even if the trainable module stays a black box.","The formulation suggests a cheaper variant: replace cosine-similarity retrieval with a learned dense retriever and keep the same joint loss, which would let retrieval errors propagate through the generator during training."],"forward_implications":["If the reported scores are representative, FinLLaMA-RAG sets a new reported baseline on the FinDER, FinQABench, FinanceBench, TATQA, and FinQA benchmarks, ahead of GPT-3 and FinBERT.","Because retrieval and generation share a joint loss, the two objectives can be traded against each other through the loss weights, letting practitioners favor retrieval precision or generation fluency for a given deployment.","The ability to aggregate cross-referenced records supports document-heavy workflows such as tax compliance, fraud investigation, and regulatory risk analysis, which the paper identifies as target applications.","The multi-hop module is dataset-agnostic, so the same architecture should transfer to other long-document domains, including legal and medical question answering."],"supporting_citations":[{"why":"supplies the FinDER dataset used for the financial QA and RAG evaluation.","marker":"[3]"},{"why":"provides the retrieval-optimization approach that the paper positions as focused on retrieval rather than generation.","marker":"[4]"},{"why":"introduces the FinTextQA long-form financial QA dataset that motivates large-context understanding.","marker":"[8]"},{"why":"presents HybridRAG, a vector-plus-knowledge-graph baseline whose simple multi-hop step the framework claims to improve on.","marker":"[9]"},{"why":"evaluates RAG systems on financial reports and documents persistent challenges with domain language and cross-document links.","marker":"[10]"}],"fun_headline_variants":["LLaMA 3 multi-hop RAG beats financial QA baselines","Multi-hop RAG with LLaMA 3 improves financial QA","Jointly trained multi-hop RAG with LLaMA 3 beats financial QA","LLaMA 3 + multi-hop RAG: higher doc-level QA accuracy","Multi-hop retrieval and LLaMA 3 raise financial QA scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the multi-hop reasoning module in Eq. (5) being a real, trainable function; if no concrete architecture, training procedure, or hop-count selection backs that module, the framework's improvement over standard RAG has no verified mechanism.","fun_headline_variants_meta":{"raw":{"variants":["LLaMA 3 multi-hop RAG beats financial QA baselines","Multi-hop RAG with LLaMA 3 improves financial QA","Jointly trained multi-hop RAG with LLaMA 3 beats financial QA","LLaMA 3 + multi-hop RAG: higher doc-level QA accuracy","Multi-hop retrieval and LLaMA 3 raise financial QA scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000978,"raw_usage":{"total_tokens":4104,"prompt_tokens":849,"completion_tokens":3255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":3157}},"tokens_in":465,"tokens_out":3255,"duration_ms":24155,"temperature":1.0,"reasoning_tokens":3157,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:14.330927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full framework and a $T=1$ variant with identical data, compute budget, and hyperparameters; if nDCG@10, BLEU, ROUGE-L, and F1 on FinDER and FinQA do not improve when $T>1$, the central claim that multi-hop reasoning drives the gains is refuted.","supporting_citations":[{"cited_title":"Evaluating retrieval- augmented generation models for financial report question and answer- ing","cited_arxiv_id":null,"evidence_quote":"evaluates RAG systems on financial reports and documents persistent challenges with domain language and cross-document links."}],"review_version":1}