{"id":"0a32b905-957d-4d7a-855d-a541ab53e03e","arxiv_id":"2512.06393","paper_version":8,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A verification-before-deduction SFT+DPO training recipe lifts accuracy on contradictory rule sets from 0% to 100% on a synthetic benchmark.","lead":"Large language models that score perfectly on simple logic puzzles drop to 0% accuracy when given contradictory rules; the authors train smaller models to explicitly check for conflicts before deducing, recovering 100% on their contradiction test. The paper also introduces a small stress-test benchmark and claims zero-shot gains on real-world natural-language inference datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Variant 3's all-false labels are an unstated normative convention; the paper's own Appendix A.6 gives a different label pattern for the same example, so V3=1.000 may measure compliance rather than logical correctness.","rationale":"The reader's weakest assumption correctly identifies the normative semantics behind the contradiction labels. My pass strengthens this: Appendix A.5 chooses an all-false 'conservative' policy, while Appendix A.6, discussing the same example, says cold and rough are supported, young is undetermined, and nice is contradicted. These are incompatible labelings, and the difference is load-bearing because the model is trained (via DPO) and evaluated on the all-false convention. The abstract additionally promises saturation on 'all four primary stress tests' for 1.5B and 8B and a Lean 4 Phase 2, none of which appear in the body; Table 2 itself reports V2=0.735, so the strong wording already exceeds the evidence. Even so, the underlying phenomenon (logic inertia) is plausible and the SFT+DPO comparison in Table 2 shows a real behavioral shift. The right response is not rejection but a condition: the authors must specify a formal semantics for contradiction cases, reconcile the A.5/A.6 conflict, and report accuracy against that semantics, ideally with the released code. That is exactly the reader's CONDITIONAL status, so no verdict change is needed.","tokens_in":13172,"tokens_out":5817,"duration_ms":57978,"concrete_test":"Re-annotate the 20 held-out Variant 3 groups using the semantics implied by Appendix A.6 ('standard conservative reasoning'): forward-chain the rules to keep derivable conclusions (Cold, Rough), leave underivable ones (Young) undetermined, and mark contradicted atoms (Nice) False; do not collapse all answers to False. Then run the released Fusion-Conflict checkpoint on this re-annotated set and compare instance-level accuracy with the paper's Table 2. If accuracy is materially below 1.000, the V3 result is an artifact of the all-false labeling protocol rather than a demonstration of logical robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive result for the central claim is Table 2's Fusion-Conflict V3=1.000. But the Variant 3 ground truth is chosen, not derived. Algorithm 1 sets A_v3 = Label(F_v3, R, Q), and Appendix A.5 states that in a contradiction 'a conservative reasoning system would recognize the contradiction and potentially reject all derived conclusions, resulting in false for all questions.' Consequently all four questions in A.4 are labelled False. The DPO preference pairs are built on exactly this convention, so the model is trained to output False whenever the verification step fires. A model that simply halts and answers False on any detected inconsistency would receive 1.000 on V3 by construction. The problem is not merely that a different (e.g., paraconsistent) semantics is possible; the paper itself is internally inconsistent. In Appendix A.6, the same Anne example is analysed as follows: 'Under standard conservative reasoning, Anne is cold and rough are supported, Anne is young is undetermined, and Anne is nice is contradicted.' That implies labels True, True, non-True, False (or T/T/undetermined/F), not False,False,False,False. Thus the benchmark's own normative analysis contradicts its labels. Since the headline 'eliminates logic inertia' rests on V3=1.000, and since the promised 8B, GPT-4o, Gemma, LIRE and RLVF results are absent from the body, the quantitative case reduces to this convention-dependent score. The paper therefore has not shown that the model reasons correctly under inconsistent premises; it has shown that it can be trained to follow one of several contestable policies.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a diagnostic benchmark of four structural stress tests (redundant/essential rule deletion, contradiction injection, logic-preserving rewrites, multi-law stacking) and documents a failure mode it calls Logic Inertia: base models achieve 1.0 accuracy on unperturbed tasks but collapse to 0.0 under injected contradictions. The authors propose Conflict-Aware Fusion, a training pipeline that enforces a 'verify premises before deducing' structural prior through SFT with a verification preamble, DPO preference pairs, and, per the abstract, LIRE regularization and RLVF with a symbolic oracle. The reported main result is that Fusion-Conflict reaches 1.000 accuracy on the base split and on Variant 3 (contradiction) while recovering 0.735 on Variant 2 (essential rule deletion), and that it transfers to LogicNLI and MNLI contradiction detection. The abstract additionally claims saturation across all four stress tests for both 1.5B and 8B backbones, as well as results for GPT-4o and Gemma-3-4B-IT, and a Lean 4 Phase 2 extension.","tokens_in":13550,"tokens_out":3864,"duration_ms":39657,"significance":"If fully substantiated, the work would be a useful contribution: a controlled perturbation benchmark that isolates different structural failure modes, and evidence that a simple verification-before-deduction training discipline can sharply improve robustness to contradictions. The availability of code and the clear reporting of baseline failures are strengths. However, several load-bearing quantitative claims are currently unsupported by the body of the paper, and the evaluation's ground-truth semantics for contradictions are internally inconsistent. The reported V3=1.000 result, which underpins the central claim of eliminating Logic Inertia, may reflect compliance with the authors' chosen labelling convention rather than correct logical behaviour under a principled semantics. The paper therefore needs substantial revisions before its central claim can be accepted.","major_comments":[{"comment":"The abstract claims the pipeline 'saturates all four primary stress tests for both 1.5B and 8B backbones' and mentions GPT-4o and Gemma-3-4B-IT results. The body only reports Fusion-Conflict results for Qwen2-1.5B and TinyLlama-1.1B in Table 2, and Table 2 reports only Base, Variant 2, and Variant 3, not Variant 1 or the Variant 4 multi-law stack. The claimed 8B, GPT-4o, Gemma-3-4B-IT, and multi-stress-test saturation are absent. This is load-bearing because the abstract's headline claim is not verifiable from the presented experiments.","section":"Abstract; §4.1, Table 2"},{"comment":"The Variant 3 ground truth is internally inconsistent. Appendix A.4/Table 7 labels all four questions False in the contradiction case, and A.5 justifies this via 'a conservative reasoning system would ... potentially reject all derived conclusions, resulting in false for all questions.' But Appendix A.6, analysing the same example, states: 'Under standard conservative reasoning, Anne is cold and rough are supported, Anne is young is undetermined, and Anne is nice is contradicted.' This implies labels True, True, undetermined/non-True, False—not False, False, False, False. Since the headline V3=1.000 is achieved by training and evaluating against the all-False convention, the benchmark's own normative analysis contradicts the labels used to score the central claim. This must be resolved, e.g., by adopting a single stated semantics and re-evaluating, or by reporting results under both the","section":"Appendix A.5 vs. A.6; Table 7"},{"comment":"The claimed causal effect of DPO is confounded by the training/evaluation loop. Stage 2 DPO preference pairs are constructed by labelling any verification-failure trace as dispreferred and any halt-with-False trace as preferred, under the same all-False convention used for V3 labels. The evaluation then reports that the model outputs the training labels. The 1.000 V3 score therefore measures adherence to the authors' protocol, not independent logical correctness. The LogicNLI/MNLI transfer in §5.5 provides one out-of-distribution check, but it is only reported as a 0.5B fine-tune and not connected to the main Table 2 pipeline with the same level of detail. To support the general claim, the authors should report held-out evaluations with alternative labelling semantics or with human-annotated/formal ground truth.","section":"§3.4, §4.4; DPO construction"},{"comment":"Stages 3 and 4 of the pipeline—LIRE regularization and RLVF with a symbolic oracle—are described in the abstract and discussed as components of the framework, but no experimental results, hyperparameters, or ablation for them appear in the body. The Lean 4 Phase 2 extension is also only mentioned in the abstract. Similarly, §5.5 states that a 0.5B model was fine-tuned with the Conflict-Aware Template and then Table 6 lists Fusion-LRA and Fusion-Conflict results, but the text only describes Stage 1 SFT, so it is unclear whether DPO/RVF were actually used for the transfer results. These missing pieces prevent the reader from assessing the full proposed method and its contributions.","section":"Abstract, §3.4, §4.2, §5.5"}],"minor_comments":[{"comment":"Algorithm 1 invokes a generic 'Label' function for A_v2 and A_v3 but does not define how ground-truth labels are computed. Please specify the labelling procedure (e.g., forward-chaining over the explicit rule set) and state whether it is the same for all variants.","section":"§3.2, Algorithm 1"},{"comment":"The naming is confusing: §3.2 defines Variant 2 as essential-rule deletion, but Appendix A.3 calls the same kind of modification 'Rule Equivalence with Different Conclusions'. Align the terminology across the main text and appendix.","section":"Appendix A.3 vs. §3.2"},{"comment":"Tables 2, 4, and 5 overlap substantially. Consider consolidating to one final-performance table with the ablation rows clearly marked, and report the number of test instances and variance/error bars; the current 20-group test set is small and no significance or confidence information is provided.","section":"§4.1, Table 2"},{"comment":"The Human Last Exam screenshot is low resolution and not accompanied by a transcript; the reader cannot verify the '0 out of 5 models answered correctly' claim. Include the exact question text, model answers, and the rubric used for correctness.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuine diagnostic idea and a plausible training recipe, but the abstract overstates what the body demonstrates. The internal contradiction between Appendix A.5 and A.6 about the ground-truth labels is a serious matter that goes beyond presentation; it affects the validity of the headline quantitative result. I would encourage the editor to require the authors to either align the benchmark labels with a single explicit semantics and re-run the evaluation, or add independent (non-training-convention) evaluations before the paper can be accepted. I do not see signs of intentional misrepresentation, but the current version is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the recipe is real; the abstract is not supported by the body. The body never delivers the abstract's four-stage pipeline, the 1.5B-and-8B saturation claim, the GPT-4o/Gemma baselines, LIRE, RLVF, or the Lean 4 Phase 2 — §4.2 says the evaluated models are BERT-base, Qwen2-1.5B, and TinyLlama-1.1B. And \"saturates all four primary stress tests\" is contradicted by Table 2's 0.735 on Variant 2; the paper's own §5.3 admits Variant 2 is the remaining weak spot. As written, the abstract describes a different paper.\n\nWhat is genuinely new: the SFT-with-verification-preamble plus DPO-preferring-halt-on-contradiction recipe, and the ablation tells a coherent story — SFT alone plateaus at 0.705 on contradiction, DPO pushes it to 1.000. The baseline diagnosis is also real and striking: 1.000 on the base template collapses to 0.000 under contradiction injection for all three baseline models. The benchmark itself is clean — one backbone, controlled perturbations. The LogicNLI/MNLI transfer numbers are the most interesting results in the paper if they survive a protocol check: a 0.5B model at 98.2% on LogicNLI from this template is either a strong result or a reporting artifact, and the paper doesn't give enough detail to tell which.\n\nThe load-bearing flaw is Variant 3's ground truth. A.4 labels all four questions False under contradiction; A.5 justifies this via \"conservative reasoning\"; A.6 then analyzes the same example and says cold and rough are supported, young is undetermined, nice is contradicted — T/T/undetermined/F, not F/F/F/F. The paper is internally inconsistent about its own normative semantics, and the DPO preference pairs bake in the FFFF choice, so the 1.000 on V3 partly measures conformity to a contested protocol. Related minor point: the synthetic test set shares the template with training, so the model might be matching surface form rather than detecting inconsistency; the transfer results are the real generalization evidence.\n\nMinor issues: 20 test groups (80 questions) per split makes \"1.0000\" a thin claim, and the LogicNLI/MNLI evaluation lacks protocol detail (split, sample size, decoding).\n\nBottom line: this deserves a serious referee, not a desk reject. The referee should require a hard revision — reconcile A.5/A.6, report or withdraw every abstract claim the body lacks, define or remove LIRE/RLVF, and provide the transfer protocol. Anyone working on reasoning robustness or DPO for reasoning should read the body, not the abstract.","headline":"A genuinely new SFT+DPO halt recipe and a striking baseline collapse, but the abstract claims results the body doesn't report and the V3 ground-truth convention contradicts the paper's own appendix.","tokens_in":14075,"tokens_out":11466,"would_cite":true,"duration_ms":96895,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a forced \"check premises before deducing\" training pipeline — a contradiction check plus preference-aligned halting on inconsistency — takes large language models from 0% accuracy under contradictory premises to 100%,","keywords":["logic inertia","contradiction detection","verification-before-deduction","structural robustness","preference optimization","cognitive structure hypothesis","rule-based reasoning","large language models"],"falsifier":"Take a contradiction-injected instance in which, by the paper's own conservative analysis, one conclusion is actually entailed by the non-contradicted premises — for example, 'Anne is cold' follows from the disjunctive fact and the green/blue rules even when a later fact contradicts 'Anne is nice' — and ask the trained model whether that entailed conclusion is true. If it answers False, the claimed 'halt-on-contradiction' is a blanket rejection rather than targeted verification, and the result would show the pipeline learned a route to high benchmark scores without the selective sensitivity th","tokens_in":13015,"feed_emoji":"🧠","tokens_out":8925,"duration_ms":84088,"temperature":0.7,"pith_summary":"The paper sets out to establish that a catastrophic failure it names Logic Inertia — LLMs persisting along learned deductive paths when premises become inconsistent — is not a scale problem but a structural one. It proposes Conflict-Aware Fusion, a training pipeline that forces models to run an explicit \"Step 1: Check Contradiction\" before applying rules, and reports that this converts both a 1.5B and an 8B backbone from 0% to 100% on contradiction injection while holding base accuracy at 1.00. The paper reads these results as support for the Cognitive Structure Hypothesis: reliable multi-step reasoning requires an explicit separation between premise verification and deductive execution. If true, it means a cheap procedural intervention, not more data, can remove a failure mode that ordinary benchmarks do not reveal.","feed_headline":"Premise-check first: LLMs go from 0% to 100% on contradiction tests","feed_subtitle":"A forced premise check turns total collapse on contradictions into perfect accuracy — and transfers to real text.","key_machinery":"The load-bearing mechanism is a dual-process structure imposed inside the chain-of-thought: the prompt template requires the model to produce a \"Step 1: Check Contradiction\" verdict before any deduction. Direct preference optimization then sharpens the decision boundary so that detected contradictions or missing premises trigger a \"Halt Reasoning\" response instead of continued inference; logical invariance regularisation penalises divergence between logically equivalent rule formulations via symmetric KL divergence; and reinforcement learning from a symbolic forward-chaining oracle rewards outputs that match deterministic entailment labels. Together these components convert premise verificat","core_discovery":"On the paper's own terms, the core discovery is that generative LLMs do not merely degrade under contradictory premises — they collapse from perfect accuracy to 0.00 on instance-level exact match when a contradictory fact is injected into an otherwise valid rule system. The paper claims Conflict-Aware Fusion eliminates this collapse: the final training configuration scores 1.000 on both the base task and the contradiction stress test, 0.735 on essential-rule deletion, and transfers zero-shot to natural-language contradiction benchmarks, reaching 98.2% and 89.4% accuracy on a first-order-logic NLI set and a multi-genre NLI contradiction subset. The authors present this as evidence that imposi","pith_inferences":["The abrupt 0% to 100% jump on contradiction tests is consistent with the model learning a conservative protocol — 'if any inconsistency is detected, answer False for everything' — rather than tracking exactly which conclusions are undermined; the paper's own worked example notes that at least one conclusion ('Anne is cold') remains supported under conservative reasoning even when 'nice' is contrad","If that is what is happening, Conflict-Aware Fusion is better described as installing a safety circuit breaker than as teaching a general logical semantics; whether that is acceptable depends on whether the deployment setting wants blanket rejection or selective inference under inconsistency.","A natural next test is to vary the required response under contradiction: training the same pipeline against a paraconsistent labeling (preserve some entailments, block the contradicted ones) would reveal whether the method can learn selective verification rather than a single halt rule.","The transfer results suggest the structural prior is domain-general, which raises the possibility of applying verification-before-deduction to argumentation and legal text, where explicit symbolic oracles are unavailable and the formal-kernel phase offers one route to trustworthy training signals."],"forward_implications":["A small model (1.5B) can be moved from total failure to perfect accuracy on contradictory rule sets by adding a mandatory verification preamble and preference pairs that reward halting — no additional data or model scale is required.","The verification prior transfers to open-domain natural-language inference, suggesting the fix is not an artifact of synthetic rule templates.","Contradiction robustness and base-task accuracy need not trade off: the final configuration holds 1.000 on the base task while fixing the contradiction collapse.","Essential-rule deletion remains only partially solved (0.735), so the method detects explicit conflicts far better than missing inference links — a clear next target.","Swapping the propositional oracle for a formally verified proof kernel reaches 99.0% agreement on classically derivable questions, pointing toward RL training with sound, verifiable labels."],"fun_headline_variants":["Contradictions crash LLMs to 0% — fusion restores 100%","Forced premise checks fix LLM total collapse on contradictions","From 0% to 100%: conflict-aware training beats logic inertia","LLMs fail contradictions unless trained with verification-first pipeline","Premise-check training turns contradictory rules into perfect accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is the paper's chosen ground truth for contradictory premises: a conservative semantics in which every derived conclusion is rejected, so all questions are labeled false. If correct behavior under inconsistency is instead classical logic (anything follows) or paraconsistent logic (some conclusions preserved), then the reported 100% accuracy measures the model's conformity to that protocol, not logical correctness.","fun_headline_variants_meta":{"raw":{"variants":["Contradictions crash LLMs to 0% — fusion restores 100%","Forced premise checks fix LLM total collapse on contradictions","From 0% to 100%: conflict-aware training beats logic inertia","LLMs fail contradictions unless trained with verification-first pipeline","Premise-check training turns contradictory rules into perfect accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1443,"prompt_tokens":909,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":653,"tokens_out":534,"duration_ms":5339,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:09:58.892789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a contradiction-injected instance in which, by the paper's own conservative analysis, one conclusion is actually entailed by the non-contradicted premises — for example, 'Anne is cold' follows from the disjunctive fact and the green/blue rules even when a later fact contradicts 'Anne is nice' — and ask the trained model whether that entailed conclusion is true. If it answers False, the claimed 'halt-on-contradiction' is a blanket rejection rather than targeted verification, and the result would show the pipeline learned a route to high benchmark scores without the selective sensitivity th","supporting_citations":[],"review_version":2}