{"id":"713eae8f-0e15-4992-bffa-04548bca74c9","arxiv_id":"2608.07838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TKFQA evaluates factuality and input-order robustness in multi-hop grounded reasoning over tables, texts, and knowledge graphs, and ORLF improves performance and order robustness on this benchmark.","lead":"This paper introduces TKFQA, a benchmark of 10,130 multi-hop question-answer pairs grounded in tables, texts, and knowledge graphs, built from counterfactual reasoning chains. It also proposes ORLF, a training framework using latent vectors and topological bias, which improves answer accuracy and order robustness on four LLM backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TKB's inter-knowledge distance and direction biases (Eqs. 7-8) are defined over the gold counterfactual chain; if the released code reads these values from test annotations, ORLF's reported EM/RCA gains and order-robustness are inflated by label leakage.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper should not be accepted as definitive. However, the single most load-bearing concern is not the correctness of the gold chains (though that matters for benchmark quality) but the possibility that ORLF's TKB module is given access to the gold counterfactual chain's topology at inference time. Eqs. (7)-(8) define the bias in terms of Dhops and I(u,v), which are example-specific quantities that, in TKFQA, are fixed by the annotated reasoning chain. The paper does not describe any mechanism by which the model could infer these quantities from the input alone; the only natural source is the dataset annotation. If the released code uses the annotation, then the comparison against baselines is systematically unfair, because ORLF receives the answer structure while the baselines do not. This would directly invalidate the headline claim of consistent gains, not just the RCA auxiliary metric. The concrete test of reading the code is decisive and cheap. I set verdict_should_be to UNCHANGED because the reader already arrived at CONDITIONAL; my concern replaces the stated weakest assumption with a sharper one but does not by itself change the final verdict. Agreement is partial because the reader mentioned TKB's dataset-specific labels only in passing and instead centered the gold-chain correctness, which I consider secondary to the possible label leakage.","tokens_in":23213,"tokens_out":8929,"duration_ms":98161,"concrete_test":"Inspect the released source code to determine, for a test example, how Dhops(u,v) and I(u,v) are obtained for Eqs. (7)-(8). If they are read from the dataset's annotated counterfactual chain (e.g., the gold chain's entity path), then ORLF uses test-time label leakage and the central comparison is invalid. As an independent check, train ORLF on a version of TKFQA where the chain direction labels are randomly permuted (e.g., Table and KG swapped in the topology), then evaluate on the unperturbed test set; if performance drops materially relative to the standard run, it confirms the model relies on the gold topology rather than learning it from the input.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines the topological knowledge bias (TKB) as b(u,v) = b_IKPD(u,v) + b_IKD(u,v) + b_SG(u,v), where b_IKPD is a learned function of Dhops(u,v), the \"number of hops between different knowledge contexts,\" and b_IKD is a learned function of the indicator I(u,v) of whether u is directed to v (Eqs. 7-8). In TKFQA, Dhops and I(u,v) are properties of the constructed counterfactual chain (e.g., Table->Text->KG or KG->Text->Table) and are example-specific. The paper states that heterogeneous knowledge contexts are constructed from counterfactual reasoning chains, thereby naturally forming cross-structure topological relations, but it never explains how the model computes Dhops or I(u,v) from the input at inference time, when the gold chain is unavailable to the model. If the implementation looks up these values from the annotated counterfactual chain for test examples, then ORLF is given the gold reasoning structure during inference, which the baselines do not receive. This would inflate the claimed 2.15% EM and 4.29% RCA improvements and the order-robustness reductions, because the model would know the true cross-context topology independent of the input order. The large drop when TKB is removed (Table 5) is consistent with TKB carrying substantial information; the unresolved question is whether that information is legitimately derived from the input or leaked from gold annotations. The paper's limitations section does not mention this dependency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TKFQA, a 10,130-example benchmark for multi-hop grounded question answering over heterogeneous knowledge contexts (tables, text, and knowledge graphs), where each example is built from an explicit counterfactual reasoning chain. The benchmark evaluates answer correctness, reasoning-chain accuracy, and robustness to the input order of the three knowledge contexts. The authors evaluate 14 open- and closed-source LLMs and report that state-of-the-art models achieve limited reasoning-chain accuracy and are sensitive to input-order permutations. They then propose ORLF, a training framework that adds knowledge-specific latent vectors, context-wise position encoding, a latent-bridge attention mask, and a topological knowledge bias, and report that ORLF improves average Exact Match by 2.15% and Reasoning-Chain Accuracy by 4.29% over baselines across four LLM backbones while reducing order-induced standard deviation. The paper includes ablation studies, a sensitivity analysis of the latent sequence length, and a public code/data release.","tokens_in":23613,"tokens_out":4949,"duration_ms":51445,"significance":"If the central claims hold, TKFQA would be a useful benchmark for factuality consistency and order robustness in multi-hop grounded reasoning, and ORLF would be a credible parameter-efficient method for improving both accuracy and robustness. The benchmark construction is thoughtful: the counterfactual chains are designed to force grounding in the provided contexts rather than parametric knowledge, and the evaluation across six input-order permutations is a genuine strength. The paper also ships its code and data, which is valuable for reproducibility. However, the current manuscript leaves a load-bearing implementation question open: the topological knowledge bias in Section 4.2 depends on quantities that are properties of the gold counterfactual chain, and the paper does not state how these quantities are obtained at inference time. In addition, the Reasoning-Chain Accuracy metric is not precisely defined, and the experimental section lacks significance tests and a clear model-selection protocol. These issues do not necessarily invalidate the contributions, but they must be resolved before the quantitative claims can be accepted.","major_comments":[{"comment":"The topological knowledge bias TKB is defined as b(u,v) = b_IKPD(u,v) + b_IKD(u,v) + b_SG(u,v), where b_IKPD is a learned function of Dhops(u,v), the 'number of hops between different knowledge contexts,' and b_IKD is a learned function of I(u,v), an indicator of whether knowledge context u is directed to v. In TKFQA, Dhops and I(u,v) are properties of the constructed counterfactual chain (e.g., Table→Text→KG) and are example-specific. The manuscript never explains how the model computes Dhops or I(u,v) from the input at inference time, when the gold chain is not available to the model. If the implementation reads these values from the annotated counterfactual chain for test examples, ORLF is given the gold reasoning structure during inference, which the baselines do not receive; this would inflate the reported EM/RCA gains and order-robustness reductions. The large performance drop when TKB is removed (Table 5) is consistent with TKB carrying substantial information, so the source of that information must be clarified. The Limitations section does not mention this dependency. Please state explicitly how Dhops and I(u,v) are computed at inference time, and if they are derived from the input, provide the exact procedure; if they are taken from test annotations, the reported results are not valid as presented.","section":"Section 4.2, Eqs. (7)-(8)"},{"comment":"Reasoning-Chain Accuracy (RCA) is defined only as the similarity between the generated chain and the provided counterfactual reasoning chain, 'with a focus on entity-level consistency between the two chains.' No exact scoring rule is given: it is not specified whether matching is exact or partial, how entity aliases and surface forms are normalized, how steps are aligned, or how a chain with the correct final answer but an incorrect intermediate entity is scored. Since RCA is a central metric in Tables 2-5 and in the abstract's headline improvement of 4.29%, the metric must be specified precisely enough to be independently computed from the released data. Please provide the exact algorithm, including any string normalization and step-alignment procedure, and report agreement with human judgments on a sample.","section":"Section 3, Evaluation Metrics"},{"comment":"The experimental protocol does not separate hyperparameter selection from test evaluation. The sensitivity analysis in Appendix B.3 varies the latent sequence length m and then states that m=16 is used in all main experiments, but the manuscript does not say which split (train, validation, or test) was used for this selection. If the test split was used to choose m or any other hyperparameter, the reported numbers are optimistically biased. Additionally, all experiments are repeated only three times and no significance tests, confidence intervals, or per-seed spreads are reported; with differences on the order of 0.1-2 percentage points (e.g., Table 4), it is unclear whether the reported improvements over TXH and PMFT are statistically reliable. Please report the model-selection procedure and add significance testing or confidence intervals for the main comparisons.","section":"Section 5 and Appendix B.3"},{"comment":"Section 5.3 states that ORLF 'outperforms all baselines across different LLM backbones,' but Table 4 shows that on Mistral-7B, TXH achieves an Avg. EM of 96.17 while ORLF achieves 96.00. The average improvement of 2.15% in Avg. EM masks per-backbone and per-metric exceptions. Please qualify the claim to reflect the per-backbone results, and report the full comparison table in the main text rather than only in the appendix.","section":"Table 4, Mistral-7B row"},{"comment":"The gold reasoning chains and answers in TKFQA are produced by an LLM-based pipeline with automatic verification and manual review, but the manuscript reports no inter-annotator agreement, no rejection rates for the automatic verification steps, and no error analysis of the manual review. Because the benchmark's validity rests on the premise that each counterfactual chain is correct and uniquely derivable, the absence of these quality statistics leaves open the possibility that a non-negligible fraction of chains are invalid or ambiguous. If so, both the conclusion that state-of-the-art LLMs are weak at reasoning-chain accuracy and the measured improvements of ORLF would be affected. Please report the verification statistics, a sample of manual-review disagreements, and any chain-validity checks.","section":"Section 3 and Appendix A.2.2"}],"minor_comments":[{"comment":"Several numeric entries lack separating spaces or commas, e.g., '87.8992.3383.67' in Table 3 and '84.6684.87 83.9085.44 86.64 85.570.85' in Table 2; these should be formatted consistently for readability.","section":"Tables 3 and 4"},{"comment":"The definition of Order Standard Deviation should state explicitly that it is computed across the six input-order permutations for a single metric (EM or RCA) and should clarify whether the values in Tables 2 and 4 are standard deviations of per-example scores or of averaged scores over the three seeds; the current wording is ambiguous.","section":"Section 3, Evaluation Metrics"},{"comment":"O. Std. values of 0.00 for EM on Llama-3.1-8B and Mistral-7B with ORLF are printed to two decimals and may be rounding artifacts; report these values with more precision or note explicitly that they are zero after rounding.","section":"Tables 4 and 5"},{"comment":"The ReAct prompt in Figure 12 tells the model that 'the known reasoning dependency is Table -> Text -> Knowledge Graph,' which gives this training-free baseline prior knowledge of the gold topology. This should be discussed as a potential advantage for ReAct and, ideally, controlled by also evaluating ReAct without that hint.","section":"Appendix E.4"}],"recommendation":"major_revision","confidential_remarks":"The most important point to verify is the implementation of TKB in the released code. If Dhops and I(u,v) in Eqs. (7)-(8) are read from the test annotations at inference time, the central empirical claim of the paper is not valid as reported and the paper would need to be re-evaluated after removing the leakage. I would recommend asking the authors to state the inference-time computation explicitly and to make the relevant code path available before the manuscript is accepted. The benchmark itself, modulo the missing verification statistics, is a reasonable contribution to the area."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper builds a real new resource. TKFQA gives you 10k QA pairs over tables, text, and KGs, each with a counterfactual reasoning chain and six input-order permutations. That combination is new, and it lets you measure something—order-robust grounded reasoning—that prior benchmarks don't jointly capture. The authors also evaluate 14 open and closed LLMs and show, convincingly, that reasoning-chain accuracy is far below answer accuracy and that order matters. That part is honest and useful.\n\nORLF is also a genuine attempt: latent vectors per knowledge type, context-wise position encoding, a latent-bridge attention mask, and a topological bias. The components are inventive and the ablation shows each contributes. I believe the method does what it says, and the gains over LoRA baselines are plausible.\n\nNow the soft spots. The RCA metric is underspecified. The paper says it measures entity-level consistency between generated and gold chains, but gives no exact scoring rule—exact match? F1? ordered or unordered? That needs to be pinned down or the headline 4.29% improvement is not interpretable. There are no significance tests anywhere; with three seeds and differences of a few points, that matters. The 0.00 O.Std values in Table 4 for EM on two backbones look like rounding artifacts at best, and at worst suggest the decoding wasn't actually stochastic as described. The gold chain construction pipeline reports no inter-annotator agreement or rejection rates, so we can't judge how noisy the benchmark labels are.\n\nThe bigger concern is TKB. The paper defines Dhops and I(u,v) over the counterfactual chain but never says how the model gets these values at inference. If the code reads them from the test annotations, ORLF is being handed the gold reasoning topology, and its EM/RCA gains and order-robustness are inflated. I checked the appendix: the standard prompt template in E.2 includes the reasoning direction as a header (\"Table->Text->KG\"), so it's possible the direction is available to all methods in the prompt and TKB is just using that input legitimately. But the paper doesn't say that, and the TKB equations are defined on the dataset topology, not on the prompt text. The authors need to state explicitly how Dhops and I(u,v) are computed at inference, and the released code needs to show it. Until then, the central comparison is under a cloud.\n\nBottom line: this paper deserves a serious referee. The benchmark is a contribution, the method is worth examining, and the empirical claims are defensible if the TKB question resolves. Send it to review, but the referee should demand a precise RCA definition, significance testing, and a clear statement about TKB's inference-time inputs. I'd cite the benchmark in my own work; it fills a real niche.","headline":"TKFQA is a genuinely useful new benchmark and ORLF is clever, but the experimental reporting has gaps (underspecified RCA, no significance tests, unresolved TKB input question) that make the headline gains provisional.","tokens_in":24095,"tokens_out":4068,"would_cite":true,"duration_ms":46254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A counterfactual benchmark across tables, texts, and knowledge graphs shows LLMs' final answers often outrun their reasoning chains, and a new latent-fusion training method narrows that gap by 4.29% in chain accuracy while cutting…","keywords":["grounded reasoning","counterfactual benchmark","heterogeneous knowledge","order robustness","factuality consistency","multi-hop QA","parameter-efficient fine-tuning","knowledge graphs"],"falsifier":"Re-annotate a random sample of TKFQA's 10,130 QA pairs with independent human annotators blind to the original labels, recording rejection rates and inter-annotator agreement on chain correctness; if a substantial fraction of gold chains are judged invalid or ambiguous, the benchmark's claims about state-of-the-art weakness and ORLF's improvements collapse. Alternatively, recompute Reasoning-Chain Accuracy with exact entity-sequence matching instead of LLM-based similarity to see whether the 4.29% gain survives.","tokens_in":23057,"feed_emoji":"🧠","tokens_out":8797,"duration_ms":75677,"temperature":0.7,"pith_summary":"The paper introduces TKFQA, a benchmark of 10,130 question-answer pairs that force large language models to reason stepwise across tables, text passages, and knowledge graphs. Each question is built from a deliberately counterfactual reasoning chain so that a model cannot rely on memorized facts; it must ground every hop in the supplied context. Evaluating 14 state-of-the-art models, the paper finds they often produce a correct final answer while failing to reproduce the correct reasoning chain, and their accuracy swings with the order in which the three context types appear. To fix this, the paper proposes ORLF, a light-weight training framework that adds trainable latent vectors, context-wise position encoding, a latent-bridge attention mask, and a topological knowledge bias, and reports that ORLF improves average exact-match accuracy by 2.15% and reasoning-chain accuracy by 4.29% across four backbone LLMs while shrinking order-induced variance.","feed_headline":"LLM chain accuracy up 4.29% via order-robust training","feed_subtitle":"A counterfactual benchmark shows LLMs often answer right for wrong reasons; ORLF restores chain fidelity.","key_machinery":"The load-bearing mechanism is the counterfactual reasoning chain $\\tilde{C}=\\langle \\tilde{T}, \\tilde{P}, \\tilde{G}\\rangle_{(e_o,e_{ta},e_{te},e_{kg})}$ that links the three knowledge modalities through replaced entities, making memorized knowledge useless and turning the task into a topological multi-hop traversal. On the training side, ORLF's engine is the combination of three interventions on the backbone's attention: context-wise position encoding (CPE) resets position IDs inside each context so order permutations are less disruptive; a latent-bridge attention mask (LBAM) allows trainable latent vectors $z_c$ to attend across contexts while ordinary tokens stay within their own context; and topological knowledge bias (TKB) injects a learned per-layer, per-head bias composed of inter-knowledge path distance, directional relation, and a Magnetic Laplacian spectral-geometry term. Together they let the model preserve knowledge-specific bias and encode cross-context topology without touching the frozen backbone weights.","core_discovery":"The central claim is that factual consistency and order robustness in multi-hop grounded reasoning are separable, measurable, and trainable: a model can be made to internalize the topological structure of heterogeneous knowledge contexts rather than treating them as an unordered bag of tokens. TKFQA operationalizes this by annotating each QA pair with a counterfactual chain through a table, a text, and a knowledge graph, and scoring models on exact match, reasoning-chain accuracy, and the standard deviation of both metrics across all six context orderings. ORLF is the paper's answer: it inserts knowledge-specific trainable latent vectors into the input sequence, resets positional indices within each context (CPE), lets latent vectors bridge attention across contexts while confining original tokens to their own context (LBAM), and adds a learned bias encoding inter-context distance, direction, and spectral geometry (TKB). The result, across Qwen3-8B, GLM-4-9B, Llama-3.1-8B, and Mistral-7B, is consistent improvement over training-free and LoRA baselines on both accuracy and stability.","pith_inferences":["A natural extension is to replace relations rather than entities along the counterfactual chain, which would test whether models track the structure of the knowledge graph, not just the identity of nodes.","Because ORLF requires access to the internal attention mechanism and positional encoding, the method cannot transfer to API-only models; distillation or order-calibration prompting would be the only route for closed-source systems, a consequence the authors state as a limitation but do not explore.","The reported gains could be stress-tested by evaluating on random orderings beyond the six permutations or with interleaved context blocks, which would separate genuine order-robustness from template-level serialization effects.","The counterfactual construction could be ported to other modalities (images, audio, code) to build adversarial benchmarks that prevent models from falling back on parametric knowledge."],"forward_implications":["If TKFQA is a fair test, then current state-of-the-art LLMs cannot be trusted to produce faithful reasoning when grounding in heterogeneous sources; answer accuracy overstates reasoning quality.","If ORLF's improvements generalize, parameter-efficient fine-tuning with structure-aware attention can make open-weight LLMs substantially more reliable for retrieval-augmented and knowledge-grounded applications without closed-source API access.","Reasoning-chain accuracy, not just final-answer EM, should become a standard reporting metric for grounded QA, since final answers can be right for the wrong reasons.","Input-order robustness should be reported as O. Std. across context permutations; the paper shows this variance is large and reducible."],"supporting_citations":[{"why":"Supplies the Wikidata KG that TKFQA's source data and KG subgraphs are drawn from.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"HybridQA's table-text multi-hop QA paradigm is the scaffold TKFQA extends with KGs and counterfactual chains.","marker":"Chen et al., 2020"},{"why":"CofCA's step-wise counterfactual QA design motivates the counterfactual reasoning chains used for grounding.","marker":"Wu et al., 2025"},{"why":"FACTS defines the factuality-grounding evaluation that TKFQA targets and extends to heterogeneous contexts.","marker":"Jacovi et al., 2025"},{"why":"PMFT supplies the context-permutation analysis of order sensitivity and the LoRA-based baseline ORLF must outperform.","marker":"Huang et al., 2025"},{"why":"RoPE is the rotary position encoding that CPE modifies by resetting position indices per context.","marker":"Su et al., 2024"},{"why":"Transformer-XH provides the extra-hop attention mechanism behind the TXH baseline.","marker":"Zhao et al., 2020"},{"why":"ReAct is a training-free reasoning-and-acting baseline ORLF is compared against.","marker":"Yao et al., 2023"},{"why":"CoT-D is a training-free decoding-time reasoning baseline ORLF is compared against.","marker":"Wang and Zhou, 2024"}],"fun_headline_variants":["ORLF boosts LLM reasoning-chain accuracy by 4.29%","Counterfactual benchmark exposes LLM reasoning gaps; ORLF fixes them","Train LLMs to reason robustly across mixed knowledge contexts","New training method improves LLM reasoning fidelity despite input order","TKFQA benchmark IDs LLM reasoning chain gaps; ORLF training fills them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold reasoning chains and answers in TKFQA are correct and uniquely derivable; the construction pipeline relies on LLM verification and manual review without reporting inter-annotator agreement or rejection rates, so if a non-negligible fraction of chains are invalid or ambiguous, both the diagnosis of state-of-the-art weakness and ORLF's measured gains lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["ORLF boosts LLM reasoning-chain accuracy by 4.29%","Counterfactual benchmark exposes LLM reasoning gaps; ORLF fixes them","Train LLMs to reason robustly across mixed knowledge contexts","New training method improves LLM reasoning fidelity despite input order","TKFQA benchmark IDs LLM reasoning chain gaps; ORLF training fills them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2923,"prompt_tokens":1045,"completion_tokens":1878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":1786}},"tokens_in":661,"tokens_out":1878,"duration_ms":13585,"temperature":1.0,"reasoning_tokens":1786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:47:16.970163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of TKFQA's 10,130 QA pairs with independent human annotators blind to the original labels, recording rejection rates and inter-annotator agreement on chain correctness; if a substantial fraction of gold chains are judged invalid or ambiguous, the benchmark's claims about state-of-the-art weakness and ORLF's improvements collapse. Alternatively, recompute Reasoning-Chain Accuracy with exact entity-sequence matching instead of LLM-based similarity to see whether the 4.29% gain survives.","supporting_citations":[],"review_version":1}