{"id":"ce38fb8f-ead7-490c-98fb-99b277332eca","arxiv_id":"2502.18467","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A small Codeforces-based study reports that DeepSeek R1 generated accepted Python solutions in fewer attempts than ChatGPT o1, while ChatGPT produced fewer static-analysis issues and shorter code.","lead":"This paper compares ChatGPT o1 and DeepSeek R1 generating Python solutions to programming problems, using online judge verdicts and code-quality metrics. It reports that DeepSeek solved more problems in fewer attempts, while ChatGPT produced cleaner and shorter code, a data point for developers choosing coding assistants.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The seven-problem Codeforces set was selected only after manual cases showed no difference, so the correctness claim rests on an outcome-dependent sample; even within that sample, 6/7 vs 3/7 gives McNemar p=0.25.","rationale":"The reader's weakest assumption is correct and is the most load-bearing issue. The paper's own methodology admits the benchmark was adopted after manual cases showed no difference, so the selection process is confounded with the outcome. Even if Table I is taken at face value, seven paired problems cannot support a general model-superiority claim: the 6/7 vs 3/7 split has exact McNemar p=0.25, and the paper provides no uncertainty quantification. No prompts or artifacts are released, so the comparison is not independently reproducible. The correctness metric in Section IV.B cannot be verified because Table I is garbled and the reported ChatGPT value 0.1875 does not match the table's attempts if all three accepted rows are counted as one attempt each. These are not disagreements with external consensus; they are failures of the evidence to support the universal claim. The reader's REJECT verdict is therefore justified, and no adjustment is needed.","tokens_in":6889,"tokens_out":6787,"duration_ms":65024,"concrete_test":"Run a pre-registered replication on a random sample of Codeforces tasks, for example all Div. 2 A-C problems from a fixed month or 30 problems drawn before any pilot, using the same prompt template for both models and recording all attempts and verdicts. Compute exact McNemar on accepted-at-any-attempt and a paired test on attempt counts. If DeepSeek's advantage is not significant or does not replicate on the held-out sample, the general correctness claim fails. As a cheaper analytical check, recompute McNemar from Table I's verdicts: discordant pairs 3 vs 0 give p=0.25, so the reported data are already insufficient to support the general claim.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing condition for the abstract and Section VI claim is that the correctness comparison is a fair, representative measurement. Section III (Test cases) says the study switched to Codeforces only after manually created test cases failed to reveal significant differences in the generated code. The seven retained problems are therefore not a random or pre-registered sample; the evaluation was moved to a setting in which differences appeared, and no evidence is given that the problems were selected independently of the observed outcome. The sample size is also tiny: Table I has paired outcomes for seven problems. Counting Accepted at any attempt gives DeepSeek 6/7 and ChatGPT 3/7, with discordant pairs 3-0; the exact two-sided McNemar p-value is 0.25, so even the reported numbers do not reach conventional significance. The paper reports only raw ratios (0.5454 vs 0.1875) with no confidence intervals or statistical test. Section V admits missing data sources and resource constraints, and no prompts or artifacts are released. Consequently the data support at most a narrow observation about these seven tasks, not the broad conclusion that DeepSeek consistently exhibited superior performance in generating correct code.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares ChatGPT (o1) and DeepSeek (R1) on Python code generation. Correctness is evaluated by submitting generated solutions to Codeforces online judge problems, with up to three attempts per problem on a set of seven contest tasks. Code quality, efficiency, and conciseness are evaluated on ten simple programming tasks using Pylint/Flake8, execution time and memory, and active lines of code. The paper reports that DeepSeek achieves higher correctness (0.5454 vs. 0.1875 on a custom metric), often on the first or second attempt, while ChatGPT produces fewer lint issues, shorter code, and comparable or slightly better memory/time usage. The conclusion states that DeepSeek exhibits superior correctness in Python code generation and suggests an advantage in algorithmic problem-solving.","tokens_in":7091,"tokens_out":3999,"duration_ms":34821,"significance":"If the central claim were well supported, a head-to-head comparison of OpenAI o1 and DeepSeek R1 on algorithmic code generation would be useful for practitioners and researchers. The use of an external judge (Codeforces) for correctness is a sound idea, and the paper reports detailed per-problem outcomes in Table I, which is commendable. However, the study as presented does not provide a valid basis for the claimed general superiority. The benchmark was selected after preliminary tests failed to show differences, the sample is tiny, no statistical testing is performed, and the definitions of the evaluation metrics are problematic. The paper is therefore not yet a reliable empirical comparison, despite the transparency of its raw results.","major_comments":[{"comment":"The selection of the Codeforces test set is outcome-dependent. The manuscript states that manually created test cases 'failed to reveal significant differences' and that the study consequently switched to Codeforces, which 'effectively highlighted the differences.' This means the seven retained problems were not chosen independently of the observed results, so the correctness comparison cannot support a general claim about model superiority. No evidence is given that the seven problems are representative of Python coding tasks or algorithmic problem-solving more broadly.","section":"III (Test cases)"},{"comment":"The correctness evidence is statistically weak. With seven paired problems, counting an Accepted verdict at any attempt gives DeepSeek 6/7 and ChatGPT 3/7, with three discordant pairs all favoring DeepSeek; the exact two-sided McNemar p-value is 0.25. The paper instead reports a custom metric, the ratio of accepted attempts to total attempts (0.5454 vs. 0.1875), which is not a per-problem success rate and is directly affected by the arbitrary three-attempt cap. No confidence intervals or significance tests are provided, so even the direction of the effect is not reliably established.","section":"Table I and Section IV-B, Eq. (1)"},{"comment":"The efficiency metric is not meaningful as defined. Efficiency = sum of accepted attempts divided by sum of execution times for accepted attempts gives units of attempts per millisecond and mixes counts with durations. It is also computed only over accepted attempts, which creates a selection bias that is not discussed. The subsequent claim in the abstract and Section VI that 'both models showed almost similar efficiency' is based on Table II of ten simple tasks, not on the Codeforces problems used for the correctness comparison, yet the conclusion combines these different evaluations without acknowledging the mismatch.","section":"IV-B, Eq. (2)"},{"comment":"The paper admits lacking 'detailed information regarding resource constraints, server issues for DeepSeek, and data sources,' but these omissions are load-bearing for an empirical comparison. Without the exact prompts, model API settings (temperature, top-p, seed), date of runs, and the full set of generated code and judge verdicts, the experiment cannot be reproduced or independently verified. The absence of these artifacts is particularly serious given the small sample size and the outcome-dependent test-case selection.","section":"V (Discussion and Limitations)"}],"minor_comments":[{"comment":"The table header contains spelling errors ('REGARIDG', 'DeekSeek') and the column alignment is confusing because the 'Attempt 1/2/3' labels are not visually tied to the per-model columns; please reformat for clarity.","section":"Table I"},{"comment":"The entry '0..9' for DeepSeek's reverse linked list time is a typo and should be '0.9' or similar; also, the issue counts are not defined (what counts as one 'issue' in Pylint/Flake8?).","section":"Table II"},{"comment":"The figure label says 'comparion' and the bars are not annotated with numerical values or error bars; please correct the label and add the underlying numbers.","section":"Fig. 1"},{"comment":"Table III is a subjective feature comparison (e.g., 'Advanced contextual understanding') that is not derived from the experimental results; please mark it clearly as an interpretive summary or remove it.","section":"IV-C and Table III"},{"comment":"Reference [5] contains placeholder DOI text ('10.1145/nnnnnnn.nnnnnnn') and the reference list should be checked for formatting consistency.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper is an early-stage empirical study with a plausible idea but severe methodological weaknesses. The central claim of DeepSeek's superior correctness rests on an outcome-driven benchmark selection and a sample of seven problems whose paired difference does not reach statistical significance. The manuscript also lacks the reproducibility artifacts that are now standard for LLM code-generation benchmarks. While the raw data in Table I are transparent, the conclusion as written is not supported by the evidence. A revision would require essentially a new study with a pre-registered or independently sampled problem set, more problems, proper statistical tests, and released prompts and configurations, which goes beyond what a revision could patch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take. This is a small head-to-head that does not support its abstract. The only new thing is the specific pair (ChatGPT o1 vs DeepSeek R1) on a handful of Codeforces problems. That is a legitimate measurement, but it is too thin and too outcome-dependent to justify the general claim that DeepSeek is superior at Python code generation.\n\nWhat it does well: it's readable, the repair-loop protocol (submit, feed errors back, up to three attempts) is sensible, and the author cites relevant prior work (EvalPlus, DevGPT, the Copilot comparisons). The limitation note in Section V is honest about missing data and server issues. The raw result — 6/7 accepted for DeepSeek vs 3/7 for ChatGPT on this set — is what it is.\n\nThe soft spots are load-bearing. Section III says manual test cases failed to show differences, so the study moved to Codeforces and then kept seven problems. That means the benchmark was chosen after the fact because it produced differences. No pre-registration, no independence, no rationale for the seven. And even on those numbers, the discordant pairs are 3-0, so the two-sided McNemar p-value is 0.25 — not significant. The correctness metric is a raw ratio with no uncertainty. Table I is garbled, Table II mixes ten trivial tasks into a table that is supposed to show code quality, and there are no prompts or artifacts, so no one can reproduce or extend it. The conclusion in Section VI far outruns the evidence.\n\nWho is this for? Maybe someone compiling anecdotal comparisons of coding assistants, or a class discussion on benchmark design. It is not a reliable basis for choosing a tool. The author seems honest about limitations, and the thinking is not incoherent, but the central claim just isn't supported.\n\nMy recommendation: I would not send this to a serious peer review as is. It deserves to be reformulated as a small replication note with narrow claims, or rejected. If you're tracking this space, treat it as a cautionary example of outcome-dependent sampling, not as evidence.","headline":"A thin, outcome-selected benchmark whose headline claim about DeepSeek's superiority is not supported by its own numbers.","tokens_in":7601,"tokens_out":2478,"would_cite":false,"duration_ms":24138,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that DeepSeek R1 outperforms ChatGPT o1 at producing correct Python code on algorithmic contest problems, often on the first try, while ChatGPT writes shorter, cleaner code.","keywords":["ChatGPT vs DeepSeek","AI in software development","AI in code generation","Automated code generation","DeepSeek in Python","large language models","Codeforces","code correctness"],"falsifier":"Submit both models to a fixed, pre-registered set of 50 Codeforces problems of mixed difficulty with the same three-attempt judge loop; if DeepSeek's Accepted rate and average attempts-to-accept are not better than ChatGPT's across the full set, the paper's central claim fails. A smaller check is to rerun the exact seven problems across many fresh sessions, since the current table reports single sessions.","tokens_in":6661,"feed_emoji":"🤖","tokens_out":7329,"duration_ms":63606,"temperature":0.7,"pith_summary":"The paper is trying to establish which of two AI assistants writes better Python code, by comparing ChatGPT o1 with DeepSeek R1 on online-judge problems. Its reported result is that DeepSeek produces correct solutions more often, frequently on the first try, so its aggregate correctness score is higher. A reader should care because this is the judgment a developer makes when choosing a coding assistant: correctness with fewer repair cycles saves time, even when the other model writes cleaner and shorter code. The paper also reports that ChatGPT's code had fewer lint issues, comparable or slightly lower execution time and memory, and fewer active lines.","feed_headline":"DeepSeek R1 beats ChatGPT o1 at contest Python","feed_subtitle":"On seven Codeforces tasks it needed fewer attempts to hit 'Accepted', while ChatGPT kept an edge in style and brevity.","key_machinery":"The central mechanism is an iterative online-judge loop: submit generated code to Codeforces, feed a non-Accepted verdict back to the model as an error message, resubmit up to three times, and score correctness as accepted attempts over total attempts. Correctness is scored as $\\frac{\\sum C_i}{\\sum T_i}$, the number of Accepted verdicts over total submissions, giving 0.5454 for DeepSeek and 0.1875 for ChatGPT. The comparison also uses Pylint/Flake8 issue counts, measured wall-clock time and memory, and active line count, with the correctness loop carrying the paper's main argument.","core_discovery":"The paper's central discovery is that DeepSeek R1, when asked to solve Codeforces-style algorithmic problems in Python, achieved Accepted verdicts more often and with fewer resubmissions than ChatGPT o1. On the seven retained contest problems, DeepSeek solved cases that ChatGPT could not fix in three attempts, and the paper's aggregate correctness scores put DeepSeek at 0.5454 versus 0.1875 for ChatGPT. The paper also reports that on ten simple coding tasks ChatGPT posted fewer lint issues, shorter active code, and comparable or slightly lower time and memory, so the advantage is not a clean sweep. In the author's framing, DeepSeek has a potential advantage in tasks demanding precise algorithmic implementation, while ChatGPT remains competitive on style and conciseness.","pith_inferences":["Editorial extension: because the seven-problem correctness set was chosen only after manual tests failed to differentiate the models, the numeric gap should be treated as a hypothesis-generating signal until a larger pre-registered sample reproduces it.","Editorial extension: the practical gap may be larger than raw judge outcomes suggest, because a first-try Accepted solution saves developer time and re-prompting effort that a third-try fix does not.","Editorial extension: the same protocol on non-contest or non-Python tasks, or on newer model versions, could plausibly reverse the ranking, so the result is scoped to algorithmic Python contest problems."],"forward_implications":["DeepSeek's higher correctness score means an algorithmic coding workflow with judge feedback would need fewer model round-trips.","ChatGPT's lower lint-issue counts and shorter code remain advantages once correctness is not the bottleneck.","Both models can use judge error messages to repair their code, but DeepSeek converted those messages into Accepted solutions more often in this study.","The correctness and style comparisons rest on different problem sets: seven Codeforces problems for correctness, ten simpler problems for style, time, memory, and lines.","The reported efficiency scores, 0.62 for DeepSeek versus 0.51 for ChatGPT, indicate that the advantage persists when time to an accepted solution is factored in."],"supporting_citations":[{"why":"Supplies the prior result that ChatGPT (GPT-4) solves 71.875% of Leetcode problems, giving the study its ChatGPT baseline.","marker":"[2]"},{"why":"Motivates the study's use of expanded, rigorous test-case evaluation and shows that naive correctness benchmarks can mis-rank LLMs.","marker":"[4]"},{"why":"Provides prior comparative code-quality numbers for ChatGPT versus other assistants, anchoring the quality metrics used here.","marker":"[8]"},{"why":"Establishes DeepSeek as a code-specialized model family competitive with closed-source models, motivating why DeepSeek is the comparison target.","marker":"[10]"}],"fun_headline_variants":["DeepSeek R1 tops ChatGPT o1 in contest Python accuracy","DeepSeek wins Python correctness, ChatGPT wins style","Study: DeepSeek R1 better at algorithmic code generation","Code face-off: DeepSeek R1 beats ChatGPT o1 on Python","DeepSeek R1 excels at contest Python, ChatGPT stays neat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the seven Codeforces problems kept for the study represent Python coding well enough to back a general claim that DeepSeek is better; the benchmark was narrowed after manual tests produced no difference, so those seven problems could be precisely the ones where DeepSeek happens to shine.","fun_headline_variants_meta":{"raw":{"variants":["DeepSeek R1 tops ChatGPT o1 in contest Python accuracy","DeepSeek wins Python correctness, ChatGPT wins style","Study: DeepSeek R1 better at algorithmic code generation","Code face-off: DeepSeek R1 beats ChatGPT o1 on Python","DeepSeek R1 excels at contest Python, ChatGPT stays neat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3005,"prompt_tokens":920,"completion_tokens":2085,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1996}},"tokens_in":536,"tokens_out":2085,"duration_ms":13678,"temperature":1.0,"reasoning_tokens":1996,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:25:18.850217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Submit both models to a fixed, pre-registered set of 50 Codeforces problems of mixed difficulty with the same three-attempt judge loop; if DeepSeek's Accepted rate and average attempts-to-accept are not better than ChatGPT's across the full set, the paper's central claim fails. A smaller check is to rerun the exact seven problems across many fresh sessions, since the current table reports single sessions.","supporting_citations":[{"cited_title":"Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation,","cited_arxiv_id":null,"evidence_quote":"Motivates the study's use of expanded, rigorous test-case evaluation and shows that naive correctness benchmarks can mis-rank LLMs."}],"review_version":1}