{"id":"d45e6828-3e0d-44d3-9140-db7d45c4e800","arxiv_id":"2506.23128","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DeepSeek-R1 outperforms GPT-4o and DeepSeek-V3 on family tree and graph reasoning benchmarks at sizes 10 and 20, but all models collapse at size 40.","lead":"This paper compares three large language models, DeepSeek-R1, DeepSeek-V3, and GPT-4o, on family tree and graph reasoning puzzles, reporting that DeepSeek-R1 scores highest on small and medium sizes while all models fail on the largest size. It is a useful empirical check on how far current reasoning-oriented models go beyond pattern matching, but the comparisons lack statistical grounding and the code and data are not yet public.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I contradicts the claim that DeepSeek-R1 consistently achieves the highest F1-scores: at n=40, Shortest(x,y), DeepSeek-V3 (0.238) outperforms DeepSeek-R1 (0.201).","rationale":"The reader's verdict is CONDITIONAL, citing unverified ground truth, missing code/data, and format-scoring issues. Those are legitimate, but the most immediate and decisive problem is internal: the paper's own table contradicts its central claim. This is not a matter of external data or a bug that could be hidden in unreleased code; it is visible in the printed results. It directly undermines the abstract's 'consistently' and Section V-A's explicit assertion. The paper otherwise has merit: the benchmark design is clear, the zero-shot setup is reasonable, and the qualitative CoT excerpt illustrates the intended reasoning behaviors. However, a central claim that is contradicted by the paper's own evidence cannot be accepted without revision. The fix is straightforward (narrow the claim or correct the data), so CONDITIONAL remains appropriate rather than REJECT. I disagree with the reader that ground-truth correctness is the single weakest assumption; the internal inconsistency is checkable now and is more load-bearing for the paper's headline.","tokens_in":14706,"tokens_out":8579,"duration_ms":81454,"concrete_test":"Locate Table I, row labeled 'Shortest(x,y)' under the n=40 column group. Compare the entries for DeepSeek-V3 (0.238) and DeepSeek-R1 (0.201). If these values are correct as printed, then the sentence in Section V-A stating that DeepSeek-R1 consistently achieves the highest average F1-scores across all n is false; the authors should either correct the claim to acknowledge the n=40 Shortest exception or provide corrected data if the table contains a typo. Additionally, if the raw per-run scores are available, recompute the mean and standard deviation for that cell to confirm the ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and repeated in Section V-A, is that DeepSeek-R1 'consistently achieves the highest F1-scores across multiple tasks and problem sizes.' Section V-A further asserts that across all n=10, 20, and 40, DeepSeek-R1 consistently achieves the highest average F1-scores, except at n=40 where all three models fail completely on IsGrandson, IsAunt, and IsPaternalGreatAunt. But Table I, row 6 (Shortest(x,y)) at n=40 shows DeepSeek-V3 with 0.238 and DeepSeek-R1 with 0.201, so DeepSeek-R1 is not the top performer in that cell. This is an internal inconsistency between the reported data and the headline claim. Because the central claim is the paper's main takeaway, this overstatement is a load-bearing correctness risk: if the claim were corrected, the paper's contribution would narrow to 'generally highest at n=10 and 20, with an exception at n=40.' No statistical significance tests or error bars are provided, so it is also unclear whether the other claimed advantages exceed run-to-run variability; the concrete exception, however, is directly readable from the table.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three LLMs (DeepSeek-R1, DeepSeek-V3, GPT-4o) on two families of relational reasoning benchmarks: family-tree relations (HasSister, IsGrandson, IsAunt, IsPaternalGreatAunt) and general graph reasoning (Connectivity, Shortest). Datasets of 100 instances are generated for sizes n=10, 20, and 40, prompts are given zero-shot in JSON-request format, and performance is measured using F1-scores against ground-truth Boolean matrices. The paper claims DeepSeek-R1 consistently achieves the highest F1-scores, that all models degrade at n=40 largely due to token limits, and that qualitative analysis of R1's chain-of-thought reveals planning and verification behaviors. A public code repository is promised but not yet available.","tokens_in":14892,"tokens_out":3462,"duration_ms":38005,"significance":"If the results hold, the paper would provide a useful empirical comparison of deep-reasoning LLMs on structured relational tasks that are harder than simple arithmetic or symbolic benchmarks. The benchmark design is reasonable: target relations are defined by explicit logical formulas, prompts are zero-shot, and F1 is an appropriate primary metric for class-imbalanced matrices. The qualitative analysis of DeepSeek-R1's long chain-of-thought is a positive feature, illustrating planning and verification patterns that are rarely documented in this detail. However, the contribution is currently weakened by an internal inconsistency in the headline claim, missing statistical support, and the absence of public code/data to verify the ground-truth generation.","major_comments":[{"comment":"The central claim that DeepSeek-R1 'consistently achieves the highest F1-scores across multiple tasks and problem sizes' is directly contradicted by Table I: at n=40 for Shortest(x,y), DeepSeek-V3 scores 0.238 while DeepSeek-R1 scores 0.201. Section V-A states that at n=40 all three models fail completely only on IsGrandson, IsAunt, and IsPaternalGreatAunt, but this omits the Shortest(x,y) exception. The claim should be corrected to specify that R1 is highest at n=10 and n=20 on most tasks, with explicit exceptions at n=40, or the abstract and Section V-A should be revised to match the reported data.","section":"Table I, Section V-A"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported for any F1-scores, although the paper draws comparative conclusions such as 'DeepSeek-R1 consistently outperforms GPT-4o and DeepSeek-V3.' With 100 runs per condition, differences such as HasSister at n=10 (0.516 vs 0.542) or Connectivity at n=40 (0.070 vs 0.103) may well be within run-to-run variability. The authors should provide error bars or statistical tests (e.g., bootstrap or paired tests) to support the claimed advantage, especially for cells where the margin is small.","section":"Section IV-C, Table I"},{"comment":"The ground truth is not verifiable from the manuscript: the claim that 'All generated data are verified to ensure logical consistency, particularly for family relationships' is not accompanied by the verification algorithm, the generated datasets, or working code. The repository at https://github.com/kelvinhkcs/Deep-Relational-Reasoning is promised but not available in the preprint. Because every reported F1-score depends on the correctness of the Boolean-matrix generator and the deduction formulas, the authors should release the data-generation and verification code, or at least include a precise pseudocode description, before the empirical claims can be fully assessed.","section":"Section IV-B"},{"comment":"The paper attributes performance degradation at n=40 to 'intrinsic token limits' and to truncated reasoning, but no evidence is provided: there are no measurements of token usage, no statement of the max_tokens setting used for each model, and no systematic detection of truncation in the outputs. The explanation should be supported by comparing output token counts against model limits, or by showing that failed runs are indeed truncated rather than merely incorrect. Without this, the token-limit conclusion is an unsupported causal claim.","section":"Section V-C"},{"comment":"Assigning an F1-score of 0 to any invalid JSON or incorrectly shaped matrix conflates a formatting failure with a reasoning failure. This choice can substantially affect the reported scores, particularly for weaker models that may return malformed outputs. The paper should report format-validity rates separately from reasoning accuracy, or justify why a format failure should be treated identically to a wrong answer in a relational-reasoning evaluation.","section":"Section IV-D"}],"minor_comments":[{"comment":"DeepSeek-V3 is described as a 'smaller and more lightweight version of DeepSeek-R1'; in fact, DeepSeek-V3 and DeepSeek-R1 share a 671B-parameter MoE architecture, so this description is inaccurate and should be corrected.","section":"Section IV-A"},{"comment":"The definitions of IsAunt and IsPaternalGreatAunt include conditions such as ¬IsMother(x,y) and rely on an implicit gender assumption for 'aunt.' The paper should explicitly state that all relations are evaluated with the given gender assignments and clarify whether spouse relationships are represented or assumed.","section":"Section III-A"},{"comment":"The illustrative Boolean matrices and the natural-language prompt example in Section IV-B are not shown to be mutually consistent; for instance, the prompt lists facts such as 'P0 is P4's father' and 'P1 is P9's mother,' but the reader cannot verify that the displayed father/mother matrices encode exactly these facts. A short consistency check in the text would help.","section":"Section IV-B"},{"comment":"Reference [33] and Reference [38] appear to be the same paper ('Neural Logic Machines' by Dong et al.), but they are listed separately; please consolidate the duplicate citation.","section":"References"},{"comment":"There are several typographical inconsistencies, including 'DeepSeek–R1' with an en dash in Section V-D and 'textttConnectivity' in Section V-B; these should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant question, and the benchmark design is a step in the right direction. However, the internal inconsistency between Table I and the abstract/Section V-A is currently a blocking issue that must be fixed. The absence of code/data also makes it impossible for reviewers to independently verify the most important empirical results. I would encourage the editor to request a revised version with corrected claims, statistical support, and released code before further consideration. The paper's scope is appropriate for a CS.AI venue, though the contribution would be strengthened by a more careful discussion of how format failures affect the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clean zero-shot comparison of DeepSeek-R1, DeepSeek-V3, and GPT-4o on family tree and general graph reasoning tasks adapted from Neural Logic Machines. The specific F1 measurements at n=10 and n=20 are new, and the qualitative Chain-of-Thought trace analysis is a nice addition — the excerpt does show planning, verification, and abstraction behaviors worth discussing. The benchmark design is sound in one important respect: ground truth is generated externally from Boolean-matrix formulas, so there is no circularity.\n\nThe central problem is that the abstract and Section V-A claim DeepSeek-R1 \"consistently achieves the highest F1-scores across multiple tasks and problem sizes.\" Their own Table I contradicts this: at n=40, Shortest(x,y), DeepSeek-V3 scores 0.238 while DeepSeek-R1 scores 0.201. The Section V-A caveat only mentions the three family-tree tasks where all models get 0.000, not this graph case. So the headline claim is overstated. The corrected conclusion — R1 generally leads at n=10 and n=20, with exceptions at n=40 — is much weaker than advertised.\n\nOther soft spots are real but secondary. There are no confidence intervals or significance tests, so it is unclear whether the advantages at n=10/20 exceed run-to-run variability. Invalid JSON is scored as an F1 of 0, which conflates format compliance with reasoning ability; the paper should report parse-failure rates separately. The token-limit explanation is inferred from truncated outputs, not measured — no token counts or max_tokens settings are given. Finally, the code and data are promised but not available, so the ground-truth generator cannot be audited.\n\nThe paper deserves a serious referee, but not as-is. A revision should fix the claim, add significance tests or error bars, separate parse failures from reasoning failures, measure token usage, and release the code. With those fixes it would be a modest but honest empirical data point for anyone working on reasoning-specialized LLM evaluation. I would not cite it in its current form, and I would not make it a reading-group centerpiece, but it is a legitimate submission for a workshop or a short conference paper.","headline":"Useful narrow empirical comparison, but the paper's headline claim overstates its own Table I: at n=40 Shortest(x,y), DeepSeek-V3 beats DeepSeek-R1, so 'consistently highest' is not supported.","tokens_in":15494,"tokens_out":1661,"would_cite":false,"duration_ms":18863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeepSeek-R1 consistently outscores GPT-4o and DeepSeek-V3 on family-tree and graph relational reasoning at moderate sizes, while all models collapse at n=40 as token limits truncate reasoning and outputs.","keywords":["large language models","relational reasoning","DeepSeek-R1","chain-of-thought","family tree reasoning","graph reasoning","zero-shot prompting","F1-score"],"falsifier":"Independently reimplement the generation and deduction formulas from Section IV-B, regenerate the 100 instances per task at n=10 and n=20, and recompute F1 scores from the model outputs against those matrices; if the ground-truth labels differ from the authors' or DeepSeek-R1 no longer leads the ranking, the central claim fails.","tokens_in":14460,"feed_emoji":"🧠","tokens_out":12026,"duration_ms":109920,"temperature":0.7,"pith_summary":"This paper asks how close large language models are to genuine relational reasoning, and answers with a controlled comparison on two zero-shot benchmark families: family-tree relations and general graph properties. It claims that DeepSeek-R1, whose architecture produces long chain-of-thought traces, consistently beats DeepSeek-V3 and GPT-4o on these tasks at problem sizes n=10 and n=20, with F1 scores above 0.9 on several relations while its competitors fall near zero. It also claims that all three models collapse at n=40, and that the main cause is not logical failure but token-length limits: reasoning gets truncated and output matrices come back incomplete, which the scoring rule counts as zero. A close reading of DeepSeek-R1's traces shows real summarization, abstraction, verification, and mid-stream planning, but also incoherent or after-the-fact steps, so the paper argues high scores should not be read as proof of sound internal logic. This matters because relational reasoning is a deeper faculty than pattern matching on standard benchmarks, and this is one of the first direct measures of whether 'deep reasoning' models actually generalize there.","feed_headline":"DeepSeek-R1 wins relational-logic tests, then collapses at size 40","feed_subtitle":"Family-tree and graph tests show the long-reasoning model beats GPT-4o and DeepSeek-V3 until token limits end the run.","key_machinery":"The machinery is a pair of generated benchmarks built from Boolean matrices. Family-tree data starts with four basic predicates (IsFather, IsMother, IsSon, IsDaughter) and derives four target relations (HasSister, IsGrandson, IsAunt, IsPaternalGreatAunt) by the formal Boolean formulas in Section III; graph data starts with a directed HasEdge adjacency matrix and targets Connectivity and Shortest. Each instance becomes a shuffled natural-language prompt asking for a JSON matrix, and the paper scores the returned matrix with standard or macro-averaged F1 against the Boolean ground truth, assigning F1 0 to invalid JSON or wrong matrix shapes. This machinery lets the authors scale relational depth by n and compare models in a zero-shot setting without in-context examples.","core_discovery":"On the paper's own terms, the discovery is that DeepSeek-R1 is the only tested model that can carry out multi-step relational deduction from raw facts: at n=10 it reaches F1 0.803 on HasSister, 0.778 on IsGrandson, 0.980 on IsAunt, and 0.969 on Shortest, while GPT-4o and DeepSeek-V3 usually stay below 0.6 and often at 0.000 on the same tasks. The advantage persists at n=20 for the easier family relations but thins on graph problems, and at n=40 every model essentially fails, with all three at 0.000 on IsGrandson, IsAunt, and IsPaternalGreatAunt and graph scores at or near zero. The paper attributes the leader's edge to its long chain-of-thought style acquired through reinforcement learning, and attributes the universal collapse to token limits rather than to missing relational concepts, because failed runs show truncated reasoning and malformed JSON matrices. It also reports that the winner's reasoning traces mix structured planning and verification with mid-stream plan formation and incoherent stretches, which it takes as evidence that visible coherence can conceal unsound inference dynamics.","pith_inferences":["The scoring rule (any malformed or incomplete output equals F1 0) mixes two failure modes; a follow-up that distinguishes parse failures from completed-but-wrong answers would separate token exhaustion from actual reasoning errors.","If token limits are indeed the bottleneck, a testable prediction follows now: giving the same n=40 prompts a much larger output budget, or requiring a more compact output format, should restore nonzero F1 scores on IsGrandson and IsAunt.","Because the generated facts are shuffled and the target relations are fixed Boolean formulas, this benchmark tests execution of a rule set rather than discovery of relational semantics; varying the proof depth of each instance could map F1 against required reasoning steps.","The observed 'plan appears midway' behavior invites a controlled intervention: explicitly prompting for an upfront plan before the solution should improve accuracy if planning is causal, and should not if the plan is post-hoc narration."],"forward_implications":["Deep-reasoning architecture translates into a measurable relational-reasoning advantage at moderate scale: DeepSeek-R1's long-CoT model beats non-CoT models on almost every family-tree and graph task at n=10 and n=20.","Token budget, not relational concept knowledge, is the main barrier to scaling: incomplete and malformed outputs, scored as zero, explain much of the n=40 collapse.","The benchmark's six problems and Boolean ground-truth generation can serve as a reusable zero-shot probe for other LLMs claiming deep reasoning ability.","High F1 scores and visibly logical traces do not guarantee sound reasoning: the paper documents mid-stream planning and incoherent steps inside the winner's own chain of thought.","Problem complexity itself matters: the deepest relation tested, IsPaternalGreatAunt, holds DeepSeek-R1 to F1 near 0.37-0.39 even at n=10 and n=20, where simpler relations reach 0.9."],"supporting_citations":[{"why":"Supplies the DeepSeek-R1 model and its long chain-of-thought, reinforcement-learning design, which the paper credits for the leader's performance.","marker":"[5]"},{"why":"Provides the Neural Logic Machines benchmark family that the paper says its data generation is adapted from, so it underpins the Boolean ground-truth construction.","marker":"[38]"},{"why":"Defines chain-of-thought prompting, the reasoning mechanism the paper analyzes in DeepSeek-R1's traces.","marker":"[26]"},{"why":"Establishes the zero-shot reasoning setting that the paper uses for all prompts and comparisons.","marker":"[29]"},{"why":"Reports that relational reasoning remains a weakness of current LLMs, which motivates the benchmark and the comparison against this prior expectation.","marker":"[18]"}],"fun_headline_variants":["DeepSeek-R1 tops relational graph tests, then hits token wall","R1 reasons deep, but only up to a point — token limits bite","Long thought, short memory: DeepSeek-R1 fails big graphs","Relational reasoning: R1 leads, but all LLMs collapse at scale","DeepSeek-R1's deep reasoning collapses under token limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every F1 comparison depends on the Boolean-matrix generator and the deduction formulas in Section IV-B producing correct ground truth, but the preprint promises code without releasing it, so a bug in those formulas would silently invalidate the reported scores.","fun_headline_variants_meta":{"raw":{"variants":["DeepSeek-R1 tops relational graph tests, then hits token wall","R1 reasons deep, but only up to a point — token limits bite","Long thought, short memory: DeepSeek-R1 fails big graphs","Relational reasoning: R1 leads, but all LLMs collapse at scale","DeepSeek-R1's deep reasoning collapses under token limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3211,"prompt_tokens":1050,"completion_tokens":2161,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":2065}},"tokens_in":666,"tokens_out":2161,"duration_ms":16293,"temperature":1.0,"reasoning_tokens":2065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:48:15.846262+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently reimplement the generation and deduction formulas from Section IV-B, regenerate the 100 instances per task at n=10 and n=20, and recompute F1 scores from the model outputs against those matrices; if the ground-truth labels differ from the authors' or DeepSeek-R1 no longer leads the ranking, the central claim fails.","supporting_citations":[{"cited_title":"Llms for relational reasoning: How far are we?","cited_arxiv_id":null,"evidence_quote":"Reports that relational reasoning remains a weakness of current LLMs, which motivates the benchmark and the comparison against this prior expectation."}],"review_version":1}