{"id":"01cb6510-c87c-4bd2-b002-4deeaab91420","arxiv_id":"2505.00776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new four-variant benchmark of simple geometric logic questions shows most LLMs score near chance, with performance largely stable across small language variations.","lead":"This paper introduces a 432-question benchmark of simple geometric logic problems and tests 24 large language models on four linguistically varied versions. It finds that most models hover near random accuracy, that few-shot prompting does not help, and that answer stability across rephrasing is generally high.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The invariability test for Q1 vs Q4 is ambiguous: the paper never states whether Q4 responses were reversed before the McNemar comparison, so the reported 'invariance' could be an artifact of not accounting for the flipped correct answers.","rationale":"The reader's identified weakest assumption is the most load-bearing concern because it directly undermines the paper's novel invariability analysis, which is a core contribution as reflected in the title. The accuracy findings are likely robust, but the invariability conclusion is ambiguous without knowing whether Q4 responses were recoded before the McNemar test. My review also noted a secondary inconsistency: the abstract and introduction claim 'most models are statistically indistinguishable from the baseline,' while Section V states 'the other models are mostly statistically distinguishable from the baseline.' This is a reporting error that could mislead readers about the main accuracy result, but it is less fundamental than the Q4 ambiguity because the broader conclusion of poor reasoning does not depend on that specific phrasing. The recommended verdict remains CONDITIONAL: the authors can resolve the concern by clarifying the Q4 alignment, releasing the evaluation artifact, and fixing the wording inconsistency.","tokens_in":15347,"tokens_out":9988,"duration_ms":97922,"concrete_test":"Have the authors rerun the McNemar test for the Q1–Q4 pair after reversing Q4 responses (yes<->no) before comparison, and release the per-item raw answer arrays (108 x 4 x runs) for all models. If the p-values for models previously deemed invariant drop below the reported thresholds or the discordant balance changes sign, the invariability conclusion fails. Without such recoding, the test measures answer stability rather than semantic invariance, so the result should be reported both ways.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section V-A reports a McNemar test comparing answers between batches, including Q1 vs Q4, and concludes that models are invariant to language modifications. The paper says Q4 'only negates the question (and its answer)' and that an invariable model 'should provide the same answers in each batch.' This is internally inconsistent: negating a question flips its correct answer, so a logically consistent model should produce opposite responses on Q1 and Q4. If the authors compared raw answers without recoding Q4 (yes<->no), the test would count correct logical behavior as variability, and a stable answer pattern would actually reflect response bias rather than semantic invariance. The paper does not state whether any recoding was performed, nor are the raw answer matrices or evaluation code provided to disambiguate. Since the invariability claim is a central contribution highlighted in the title and conclusion, this ambiguity makes the claim unverifiable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new benchmark for shallow logical reasoning, consisting of 108 questions about geometric figures under four batches (Q1–Q4). Q1 is the original set; Q2 adds explicit geometric facts; Q3 replaces shape names with invented terms; Q4 negates the questions and flips the correct answers. The authors evaluate 24 LLMs under zero-shot and few-shot prompting, and 22 LLMs under two chain-of-thought variants. They report per-batch and aggregate accuracies, compare models against a 'always no' baseline, and use McNemar tests to claim that model answers are largely invariant across the four batches. The paper's central claims are that LLMs cannot yet reliably solve one-step logical deductions, that performance degrades in the few-shot setting, and that answer behavior is stable under small linguistic variations.","tokens_in":15495,"tokens_out":5762,"duration_ms":57335,"significance":"The dataset is a genuinely useful contribution: the domain-free geometric setting reduces reliance on world knowledge and memorized phrases, and the four-batch design is a sensible way to probe prompt sensitivity. The finding that many LLMs, including large ones, perform far below ceiling on these simple deductions is a meaningful, falsifiable result. The authors also make their data publicly available and report repeated runs with different seeds, which supports reproducibility of the accuracy tables. However, the invariability analysis—a central part of the title, abstract, and conclusions—is not verifiable as written because the handling of the negated Q4 batch in the McNemar comparison is ambiguous, and the filtering of tested pairs is not described. If the authors clarify the recoding and report the full test results, the paper could be a valuable empirical contribution; as it stands, the main invariability claim needs revision.","major_comments":[{"comment":"The invariability test for Q1 vs. Q4 is ambiguous and potentially internally inconsistent. The paper states that the fourth batch 'only negates the question (and its answer)' and then says that an invariable model 'should provide the same answers in each batch.' But negating a question flips its correct answer, so a logically consistent model should produce opposite yes/no responses on Q1 and Q4. If the raw Q1 and Q4 answers were compared in the McNemar test without recoding Q4 responses (yes↔no), correct logical behavior would be counted as variability. The manuscript never states whether such recoding was performed, nor are the raw answer matrices or analysis code provided. As written, the central invariability conclusion is unverifiable.","section":"V-A"},{"comment":"The procedure for selecting pairs for the McNemar test is not a valid basis for the invariability claim. The authors state that the test was applied 'only to the pairs that were not already deemed as statistically different from the accuracy analysis; all other cases clearly yield variable answers.' This ex post filtering removes precisely the pairs where answer changes are detectable and makes the subsequent 'all pairwise statistics yield a p-value over 0.1' statement uninterpretable: the set of tested pairs is not specified, and the rule itself biases the analysis toward finding invariance. The authors should report McNemar results for all pairwise batch comparisons without such filtering, or provide a pre-registered justification.","section":"V-A"},{"comment":"The paper contradicts itself about how many models are distinguishable from the baseline. The Introduction states 'Most other models are indistinguishable from the baseline,' while Section V states 'The other models are mostly statistically distinguishable from the baseline.' Table I shows that several models, such as recurrentgemma-2b (67.98 ± 2.49), Phi-3-mini (65.28 ± 2.59), and Qwen72B (70.60 ± 2.48), have confidence intervals far above the baseline of 50.93. The manuscript should state which claim is intended and support it with explicit statistical comparisons rather than relying on informal reading of confidence intervals.","section":"I and V"},{"comment":"The statistical comparisons between models rely on the intersection of 95% confidence intervals, e.g., 'its difference with Qwen72B, Llama70B, Phi-small, recurrentgemma-2b are not statistically significant (their confidence intervals intersect).' Overlapping confidence intervals do not imply a non-significant difference; this is a well-known fallacy. Formal tests (or proper simultaneous intervals) are needed for any claim of statistical equivalence or non-inferiority among models.","section":"IV and V"},{"comment":"The paper does not specify how the three runs with different seeds are used in the McNemar invariability tests. Accuracy tables report means over runs, but the McNemar test requires paired individual answers. The authors should state whether the test uses answers from a single run, a pooled set, or a majority vote, and they should provide the answer-level data to make the analysis reproducible.","section":"V-A and IV-A"}],"minor_comments":[{"comment":"In the few-shot paragraph, 'as in one-shot prompting' should be 'as in zero-shot prompting'.","section":"V"},{"comment":"The manuscript reports that 'all the pair-wise statistics obtained from the McNemar test yield a p-value over 0.1' but does not provide the actual p-values, test statistics, or the number of discordant pairs. Adding a table with these values would substantially improve the empirical support.","section":"V-A"},{"comment":"The example Q4 sentence says 'the fourth batch only negates the question (and its answer)'; this wording is confusing because negating a question does not leave the answer token unchanged. Clarifying the intended semantics and the expected response behavior would remove ambiguity.","section":"III-B"},{"comment":"Reference [31] lists the venue as 'Proc. EMLP 2023'; this appears to be a typo for EMNLP 2023. Also, the model name 'stablelm-2-1 6b-chat' in Tables I and II has an inconsistent space and should be rendered uniformly.","section":"References"},{"comment":"The text says 'in most cases, significantly' when describing CoTafter improvements, but no statistical test for the CoT comparison is described. Either add the test or soften the claim.","section":"V"}],"recommendation":"major_revision","confidential_remarks":"The invariability claim is central to the paper's title and conclusions, and the Q1 vs. Q4 comparison is the clearest test of that claim. The ambiguity about recoding Q4 responses is a blocking issue, not a presentation nit. If the authors can supply the exact comparison procedure, the answer-level data, and full McNemar results, the paper may be salvageable. The inconsistent statements about baseline distinguishability also need to be reconciled. The dataset itself and the low-accuracy finding are valuable, so I would not recommend rejection if these load-bearing points can be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the dataset and the four-batch variant design are genuinely useful: 432 controlled questions about geometric figures, testing existential/value/number restrictions and negation, with Q2 adding explicit shape facts, Q3 swapping in invented words, Q4 negating the question. Second, the paper's main empirical finding—most of 24 LLMs, including 70B ones, hover near an always-no baseline on these one-step deductions—is credible and robust across models, seeds, and prompt settings. But the title-level claim about \"invariability\" rests on an ambiguous McNemar comparison between Q1 and Q4, and that needs to be fixed before I'd trust it.\n\nWhat's actually new: the packaged benchmark and the systematic cross-model invariance measurement. The underlying logical constructs are not novel, and the authors know it; their contribution is the controlled variant set and the breadth of models. The CoT-before vs CoT-after comparison is also a nice touch—rationale before answer often hurts, rationale after often helps—and the direction is interesting even if not huge.\n\nWhere it's soft. The invariability section says Q4 \"negates the question (and its answer)\" and then asserts an invariable model \"should provide the same answers in each batch.\" Those two statements cannot both be true for Q1 vs Q4. If a model answers consistently, its literal yes/no token on Q4 should be the opposite of its Q1 token. The paper never says whether Q4 responses were recoded before the McNemar test. If they weren't, the test would classify correct logical behavior as variability, and the reported \"no significant difference\" would be consistent with models just repeating a biased answer. The raw answer matrices or evaluation code would resolve this, but the GitHub link is for the dataset, not the eval harness. This is a load-bearing ambiguity because invariability is in the title and the conclusion.\n\nThe accuracy tables themselves seem internally consistent. I don't see a circularity problem: answers are fixed, no fitted parameters. The baseline treatment is fair. The missing eval code is a real but minor transparency cost; the dataset link is there.\n\nBottom line: this paper is for anyone building or evaluating LLMs on knowledge-representation tasks. It deserves a serious referee, but the referee should ask the authors to state the Q4 alignment, release the eval code, and redo the invariability analysis accordingly. I'd take it to reading group and might cite the dataset, but not the invariability conclusion as it stands.","headline":"Useful new shallow-logic benchmark with a robust negative accuracy result, but the headline invariability claim is not verifiable as written because Q1/Q4 negation alignment is never specified.","tokens_in":15993,"tokens_out":2419,"would_cite":true,"duration_ms":23427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most large language models, including several over 70 billion parameters, perform no better than an always-'no' baseline on simple one-step logical deduction questions, and their answers barely change when the questions are reworded.","keywords":["large language models","logical reasoning","benchmark dataset","prompt dependency","invariability","chain-of-thought prompting","negation","shallow reasoning"],"falsifier":"Open the published dataset, take a model's Q1 and Q4 answer pairs, invert every Q4 answer, and rerun the McNemar test on the Q1-Q4 pair; if the p-value drops below 0.05, the invariability conclusion is reversed by the correction that the paper appears to omit.","tokens_in":15134,"feed_emoji":"🧩","tokens_out":15199,"duration_ms":129679,"temperature":0.7,"pith_summary":"This paper asks whether current large language models can perform shallow logical reasoning in a stripped-down setting: one-step deductions about geometric figures described entirely in the prompt. Across 24 models and 432 test questions, only a handful—mostly the largest ones—reach around 70 percent accuracy, while most models are statistically indistinguishable from a baseline that always answers 'no.' The paper also argues that model answers are largely invariant under small rewording of the questions, and that chain-of-thought prompting generally helps only when the rationale is requested after the yes/no answer. If the result holds, one-step deductive inference is not a reliably solved capability in current LLMs, and prompt engineering has little leverage on this class of questions.","feed_headline":"Most large language models fail one-step logic questions","feed_subtitle":"A 432-question benchmark finds only the largest models beat an always-'no' baseline, and rewording changes little.","key_machinery":"The load-bearing object is the four-batch benchmark built on 108 base questions, paired with the McNemar test, a significance test for paired binary outcomes. Each question asks for a yes/no conclusion from two premises in a geometric domain with no real-world priors; the four batches hold the underlying logical content fixed while varying surface form (extra explicit facts, invented shape names, and negation with flipped correct answer). The McNemar test is what turns 'the model answers differently on batch X than batch Y' into a statistical statement: it counts how unbalanced the discordant answer pairs are, and all comparisons in the paper come out with p-values above 0.1. The benchmark's role is to isolate the logical constructor being tested, so a failure can be attributed to reasoning over existence, universal statements, counting, or negation rather than to memorized world knowledge.","core_discovery":"On the paper's own terms, the central discovery is that shallow logical reasoning—one conclusion drawn from two premises, with no case analysis or intermediate memory—is not reliably solved by any of the 24 models tested. The dataset uses 108 base questions about geometric figures, each repeated in four batches: the original phrasing; the same phrasing with explicit facts about shapes; the same questions with invented shape names; and a negated version whose correct answer is flipped. Because 55 of the 108 base questions have 'no' as the correct answer, the baseline that always says 'no' scores 50.93 percent. In zero-shot testing, GPT-4 reaches 72.30 percent overall accuracy, but its confidence interval overlaps with Qwen1.5-72B, Llama-3-70B, Phi-3-small, and recurrentgemma-2b, and the remaining models are, with few exceptions, statistically indistinguishable from that baseline. Few-shot prompting lowers performance, leaving only GPT-4, Qwen72B, Llama70B, and Phi-small distinguishable from the baseline. For the batch pairs not already flagged as different by accuracy, pairwise McNemar tests yield p-values above 0.1, which the paper reads as invariability under language variation, and between-model answer similarity is also high. With chain-of-thought prompting, asking for the rationale before the answer pushes several models far below chance and produces many responses that are not yes/no, while asking for the rationale after the answer generally improves accuracy.","pith_inferences":["In Section V-A, batch Q4 is described as negating the question and flipping its answer, and an invariable model is said to 'provide the same answers in each batch.' The paper never states that Q4 responses were inverted before the McNemar comparison; without that inversion, logically correct flips would be counted as variability, so a reader should verify this before relying on the invariability c","The four-batch protocol could be extended to paraphrases generated by another LLM: the paper's invariability claim is about small hand-designed variants, and it is untested whether invariance survives larger surface variation.","The Q1/Q4 pair suggests a targeted probe of negation handling: compare accuracy on Q1 with accuracy on Q4 after inverting Q4 labels; a model that scores at chance on both may be systematically flipping its answers, which aggregate accuracy alone cannot reveal."],"forward_implications":["If the central claim holds, accuracy on this benchmark is a diagnostic of basic logical competence: a model that scores near the always-'no' baseline on one-step deductions cannot be trusted for knowledge-based applications that require checking consistency or certainty.","Few-shot prompting made most models worse, so adding a few examples is not a reliable remedy for this class of questions; a deployed system would need a different mechanism.","The invariability across Q1-Q4 means a failure is not a quirk of one phrasing: if a model gets a question wrong, semantically equivalent rephrasings are likely to be wrong too, which makes the error harder to avoid by prompt rewriting.","Chain-of-thought prompting should be configured to ask for the rationale after the yes/no answer; the before-answer ordering can push several models far below chance and generates many responses that cannot be parsed as yes/no.","The dataset's four batches isolate specific logical constructors, so they can be used to identify which expressions (existential, universal, number restrictions, negation) a given model handles and which it does not."],"supporting_citations":[{"why":"It supplies the cognitive-psychology precedent for using geometric-figure questions to avoid answers driven by learned world knowledge.","marker":"[15]"},{"why":"It defines the description-logic constructors (existential, value, number restrictions, and negation) that the benchmark questions are built to test.","marker":"[17]"},{"why":"It defines chain-of-thought prompting, the technique whose ordering the paper evaluates by asking for the rationale before or after the answer.","marker":"[18]"},{"why":"It documents LLMs' difficulty with negation, motivating the Q4 batch and the expectation that negated questions will degrade accuracy.","marker":"[31]"},{"why":"It gives the normal-approximation confidence intervals used to compare each model's accuracy with the always-'no' baseline.","marker":"[42]"},{"why":"It provides the paired-response significance test used to decide whether answer changes across batches are statistically meaningful.","marker":"[43]"},{"why":"It represents prior broad logical-reasoning evaluations that the paper extends by controlling prompt variation and measuring invariability.","marker":"[24]"}],"fun_headline_variants":["One-step logic trips up most LLMs","Chain-of-thought can hurt or help LLM logic","Only a few LLMs beat the always-no baseline","Rewording doesn't fix LLMs' one-step reasoning","Simple deduction: most LLMs near chance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The invariability conclusion assumes that when batch Q4 negates the question and flips the correct answer, the model's Q4 responses were reversed before comparing them with Q1 responses; if the literal answers were compared, a logically correct flip would be counted as variability, and the paper does not state that this reversal was applied.","fun_headline_variants_meta":{"raw":{"variants":["One-step logic trips up most LLMs","Chain-of-thought can hurt or help LLM logic","Only a few LLMs beat the always-no baseline","Rewording doesn't fix LLMs' one-step reasoning","Simple deduction: most LLMs near chance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002032,"raw_usage":{"total_tokens":7959,"prompt_tokens":1027,"completion_tokens":6932,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":6858}},"tokens_in":643,"tokens_out":6932,"duration_ms":55868,"temperature":1.0,"reasoning_tokens":6858,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:34:34.390116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the published dataset, take a model's Q1 and Q4 answer pairs, invert every Q4 answer, and rerun the McNemar test on the Q1-Q4 pair; if the p-value drops below 0.05, the invariability conclusion is reversed by the correction that the paper appears to omit.","supporting_citations":[{"cited_title":"Working memo ry involve- ment in propositional and spatial reasoning,","cited_arxiv_id":null,"evidence_quote":"It supplies the cognitive-psychology precedent for using geometric-figure questions to avoid answers driven by learned world knowledge."},{"cited_title":"Baader, D","cited_arxiv_id":null,"evidence_quote":"It defines the description-logic constructors (existential, value, number restrictions, and negation) that the benchmark questions are built to test."},{"cited_title":"This is not a dataset: A large negation benchmark to challen ge large language models,","cited_arxiv_id":null,"evidence_quote":"It documents LLMs' difficulty with negation, motivating the Q4 batch and the expectation that negated questions will degrade accuracy."},{"cited_title":"Dekking, A Modern Introduction to Probability and Statistics: Under - standing Why and How , ser","cited_arxiv_id":null,"evidence_quote":"It gives the normal-approximation confidence intervals used to compare each model's accuracy with the always-'no' baseline."},{"cited_title":"Note on the sampling error of the differenc e between correlated proportions or percentages,","cited_arxiv_id":null,"evidence_quote":"It provides the paired-response significance test used to decide whether answer changes across batches are statistically meaningful."}],"review_version":1}