{"id":"43b03c7d-d03f-4222-af99-3c88199b2c9d","arxiv_id":"2502.00226","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 65-problem, multi-file benchmark evaluates LLM code generation on project-style tasks and reports that top models tie near 75%, with Claude-3.5-Sonnet most consistent.","lead":"This paper introduces a benchmark of 65 multi-file software project tasks and uses it to test five frontier LLMs on code correctness and consistency. It finds that the top models score about 75% with no significant performance gap, and that Claude-3.5-Sonnet is more consistent across 32 runs per problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline consistency claim rests on a paired t-test over per-problem SDs despite the paper's own non-normality caveat, with no reported p-values, pairing details, or multiple-comparison corrections, so Claude's significant edge is not established.","rationale":"The reader's weakest assumption correctly identifies the paired t-test on non-normal per-problem SDs as the linchpin of the consistency claim. My review confirms this is the most load-bearing issue: the paper's own Section 2.5 argues for the median because SDs are non-normal, yet the significance test in Section 4.1 silently relies on a parametric paired t-test. This inconsistency is concrete and testable. The absence of reported p-values, effect sizes, and multiple-comparison corrections makes the claim impossible to verify from the manuscript alone. The benchmark construction and multi-file evaluation are real contributions and not undermined by this statistical concern, so rejection would be too harsh. The appropriate response is to require the authors to provide the full statistical details and a robustness check with non-parametric methods, which matches the reader's CONDITIONAL verdict. I therefore see no reason to change the verdict, only to sharpen the specific test that would resolve the concern.","tokens_in":9364,"tokens_out":1970,"duration_ms":21201,"concrete_test":"Recompute the consistency comparison using a paired Wilcoxon signed-rank test (or a bootstrap confidence interval for the median paired difference) on the per-problem SD vectors for all model pairs, applying a Holm-Bonferroni correction across the 10 pairwise comparisons. If Claude's advantage over each other model is no longer significant at α=0.05, or if the raw p-values are not reported, the claimed statistical significance fails. Also report the Shapiro-Wilk statistic on the paired differences to justify whether the t-test was appropriate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that Claude-3.5-Sonnet-1022 is significantly more consistent than the other models—depends entirely on a paired t-test applied to per-problem standard deviations (SDs). Section 2.5 explicitly states that SD values across problems 'often deviates from a normal distribution,' which is why the paper uses the median rather than the mean as the consistency summary. Yet the significance claim in Section 4.1 still uses a paired t-test on these SDs. A t-test on 65 paired differences can be anti-conservative when the underlying distributions are skewed or heavy-tailed, and the paper reports no diagnostics for the normality of the differences, no exact p-values, and no effect sizes. Additionally, comparing Claude against 'the rest of the models' implies multiple pairwise tests (at least four, possibly ten), but no multiple-comparison correction is mentioned. The same statistical fragility applies to the 'no statistically significant differences' claim for mean scores: with high variance across 65 problems, a non-significant t-test may simply reflect low power, not equivalence. Because the abstract and conclusions prominently feature Claude's statistically significant consistency advantage, this unsupported inference is load-bearing. The benchmark itself is a useful contribution, but the headline reliability claim is not yet substantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HackerRank-ASTRA, a benchmark of 65 multi-file, project-based coding problems (mostly front-end) with evaluation over k=32 independent runs per problem. It defines three metrics: mean score, mean pass@1, and median standard deviation of scores across problems as a consistency measure. On an initial evaluation of five frontier models, the paper reports that o1, o1-preview, and Claude-3.5-Sonnet-1022 achieve comparable mean scores around 75%, and that Claude-3.5-Sonnet-1022 shows the highest consistency (median SD=0.0497), a difference claimed to be statistically significant. The benchmark and dataset are open-sourced, and the paper includes taxonomy-level and error-analysis results.","tokens_in":9592,"tokens_out":3897,"duration_ms":35103,"significance":"If the results are substantiated, HackerRank-ASTRA is a useful benchmark contribution: it addresses a real gap by evaluating multi-file project-level tasks rather than isolated functions, and it introduces a consistency metric based on repeated runs, which is rare in code-generation benchmarks. Strengths include the release of the dataset, the relatively large k=32 per problem, the structured XML/JSON output pipeline, and the taxonomy-level breakdown. The central statistical claims, however, are currently underreported, and the headline consistency advantage is not adequately supported by the evidence presented.","major_comments":[{"comment":"The claim that Claude-3.5-Sonnet-1022's lower median SD is 'statistically significant based on the paired t-test' is not substantiated. Section 2.5 states that per-problem standard deviations 'often deviate from a normal distribution,' which is the stated motivation for reporting the median; applying a paired t-test to those same non-normal per-problem SDs is inconsistent without reporting diagnostics for the 65 paired differences. The manuscript reports no p-values, effect sizes, or confidence intervals, does not specify whether the comparison is Claude versus each other model or Claude versus the pooled rest, and gives no multiple-comparison correction. Please replace or supplement the t-test with a nonparametric paired test (e.g., Wilcoxon signed-rank) or a bootstrap/permutation procedure on the paired differences, report exact p-values and effect sizes, and justify the pairing and the correction for multiple tests.","section":"Section 4.1, Table 3"},{"comment":"The sentence 'a paired t-test reveals that, with the exception of GPT-4o-0513, the differences between model performances are not statistically significant' is ambiguous and does not report the actual test results. The abstract's claim of 'no statistically significant differences' among the top three models is not the same as the text's statement involving an exception for GPT-4o-0513. The authors should provide the full matrix of pairwise comparisons with test statistics, p-values, and a stated multiple-comparison correction; without this, neither claim can be verified, and a non-significant result with 65 problems and high variance is not evidence of equivalence.","section":"Section 4.1"},{"comment":"The manuscript says the 65 project questions are open-sourced, but it does not state whether the hidden test cases, the Docker-based evaluation harness, and the aggregation scripts are also released. Since the benchmark's value depends on independent verification of the reported scores, please explicitly enumerate what is included in the release; if the test cases and harness are not public, the leaderboard cannot be reproduced by third parties, which would undercut the benchmark's utility.","section":"Section 7 and HuggingFace link"}],"minor_comments":[{"comment":"There is a typo: 'preapred' should be 'prepared'.","section":"Section 2.1"},{"comment":"The word 'argentic' should be 'agentic'.","section":"Section 7"},{"comment":"The abstract reports Claude-3.5-Sonnet-1022's median SD as 0.0497, while Table 3 reports 0.05; please reconcile the two values or explicitly state that the table value is rounded.","section":"Abstract and Table 3"},{"comment":"The claim that 'XML demonstrated statistically significant superiority over JSON' also lacks the test used and the p-values; please report the statistical details or soften the claim.","section":"Section 4.4 and Tables 5-6"},{"comment":"The model o1 is missing from the XML and JSON format comparison tables, although it appears in the main leaderboard; please clarify why o1 is excluded.","section":"Section 4.4, Tables 5 and 6"},{"comment":"Equation (3) uses a population standard deviation with division by k rather than k-1; since k=32, the difference is minor, but the choice should be stated explicitly.","section":"Section 2.5, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is produced by HackerRank personnel and draws on HackerRank's proprietary content; the paper would benefit from a clearer statement of the relationship between the authors and the benchmark owner, and of any measures taken to avoid test-case contamination of model training data. The statistical reporting is the main gap; if the consistency and mean-score claims are re-analyzed with appropriate tests and full reporting, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the benchmark itself is a genuine contribution: 65 project-based multi-file problems with real test cases, averaging 12 input files and 84 lines of expected solution code, plus a consistency protocol (k=32, median SD) that DevEval and other multi-file benchmarks don't provide. Second, the headline claim that Claude-3.5-Sonnet is significantly more consistent than the other models is not actually established by the statistics presented.\n\nWhat's new here is the combination of multi-file project tasks with repeated-run consistency measurement, and a taxonomy breakdown that lets you see where models win or lose by subskill. The authors are also honest about scope: they openly say v1 is front-end only, that agentic and iterative approaches are not evaluated, and that the model set is small. They release the 65 problems on HuggingFace, which is a real step toward reproducibility.\n\nThe soft spots are concentrated in the significance testing. The paper itself says SD values across problems often deviate from a normal distribution, which is why they use the median. Then they run a paired t-test on those SDs to claim Claude is significantly more consistent. That's inconsistent. No p-values, no effect sizes, no multiple-comparison correction when comparing against \"the rest of the models.\" The abstract says SD=0.0497 while Table 3 rounds to 0.05; that's minor but sloppy. The \"no significant difference\" among top models for mean score is also weak evidence, since with 65 problems and high variance, a null result may just be low power.\n\nThe independence concern is real but not a fatal flaw: HackerRank curates the problems, and the authors are HackerRank employees. That doesn't make the benchmark circular, but it does mean external validation would be welcome. The evaluation harness is not released, so reproducing the main results requires reconstruction.\n\nOverall, this is a solid benchmark paper with an under-supported empirical claim. The fix is straightforward: report full statistical details, use a nonparametric test or bootstrap for the consistency comparison, and ideally release the harness. The benchmark deserves a serious referee and, with revisions, publication. I'd bring it to a reading group focused on LLM code evaluation, and I'd cite it for the benchmark design even if I treat the consistency ranking with caution.","headline":"Useful new multi-file project benchmark with a real consistency protocol, but the headline consistency claim rests on shaky statistics.","tokens_in":10148,"tokens_out":1246,"would_cite":true,"duration_ms":14108,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Claude-3.5-Sonnet-1022 is the most consistent coder in a 65-task multi-file benchmark.","keywords":["LLM evaluation","code generation benchmark","multi-file projects","model consistency","front-end development","pass@1","median standard deviation","software engineering"],"falsifier":"Recompute the consistency comparison with a paired permutation test on the 65 per-problem standard deviations (or with a Wilcoxon signed-rank test) and apply a multiple-comparison correction across the model pairs; if the corrected p-values fail to stay below 0.05 for Claude-3.5-Sonnet-1022 versus the other models, the paper's headline consistency claim would be falsified.","tokens_in":9164,"feed_emoji":"🧪","tokens_out":7693,"duration_ms":63196,"temperature":0.7,"pith_summary":"This paper introduces HackerRank-ASTRA, a benchmark of 65 project-based coding problems that require reading an average of 12 source and configuration files and editing about 2.3 files each. Its purpose is to measure not just whether large language models can solve realistic multi-file tasks, but how consistently they solve them across repeated attempts. Across 32 fresh runs per problem, o1, o1-preview, and Claude-3.5-Sonnet-1022 earn nearly identical mean scores around 75 percent, with no statistically significant difference between them. The paper's central claim is that Claude-3.5-Sonnet-1022 is the most consistent of the evaluated models, with a median per-problem standard deviation of 0.0497, and that this consistency gap is statistically significant. If true, that places reliability as a measurable axis of model quality that is separate from mean correctness.","feed_headline":"Claude-3.5-Sonnet-1022 is the most consistent coder in a 65-task test","feed_subtitle":"Top models tie near 75 percent, but Claude's run-to-run variance is the lowest and the gap is statistically significant.","key_machinery":"The load-bearing mechanism is the ASTRA evaluation protocol: a structured prompt that pairs the problem statement with all relevant project files, a format-enforcing step that asks for XML or JSON output, post-processing that repairs escaping issues, integration of the candidate solution into a Dockerized project, and execution against hidden test cases. Correctness is aggregated as the mean fraction of tests passed across k=32 fresh conversations per problem, and consistency is summarized as the median of the 65 per-problem standard deviations. The paired t-test applied to those per-problem standard deviations is what carries the statistical claim that Claude-3.5-Sonnet-1022 is more consistent than the other models.","core_discovery":"The v1 ASTRA benchmark consists of 65 multi-file, project-based problems drawn from front-end development, covering 10 primary skill domains and 34 subskills across frameworks such as Node.js, React.js, Angular.js, Django, Java Spring Boot, Ruby on Rails, and .NET. Each model receives the problem statement plus the project files, returns code in XML or JSON, and is scored by the fraction of test cases passed, with 32 independent runs per problem. On mean score, o1 (75.80%), o1-preview (75.55%), and Claude-3.5-Sonnet-1022 (75.07%) are statistically tied, while GPT-4o-0513 trails at 69.52%. On consistency, defined as the median across problems of each model's per-problem standard deviation, Claude-3.5-Sonnet-1022 scores 0.0497, compared with 0.11 for o1, 0.13 for Gemini-1.5-pro, 0.17 for o1-preview, and 0.20 for GPT-4o-0513, and the authors report that this difference is statistically significant via a paired t-test.","pith_inferences":["A natural extension would be to test whether the consistency ranking is stable under alternative dispersion measures, such as the interquartile range or the variance of scores, since the paper's own distributional caveat suggests median SD may be sensitive to the test choice.","The k=32 repeated-run design makes the benchmark expensive; if consistency differences replicate on a small subset of problems, a cheaper screening protocol could use fewer runs for models already known to be consistent.","Because the 65 problems come from a proprietary item bank, an independent replication using open-sourced problems would clarify whether Claude's low median SD is a general property or an artifact of this particular task distribution.","The format effect (XML over JSON) could itself be a useful diagnostic: models with smaller XML-vs-JSON gaps might be more robust to prompt formatting, a property worth checking when a model is deployed through varied interfaces."],"forward_implications":["Model choice for production coding should weigh consistency alongside mean correctness, since two models with the same average score can have very different run-to-run reliability.","The benchmark's sub-skill results imply that no single model dominates: Claude-3.5-Sonnet-1022 wins on API integration and database interaction, while o1 wins on form handling and pagination, so task-specific selection is warranted.","The consistent XML-over-JSON advantage across models indicates that output format is a practical lever in evaluating or deploying code LLMs on multi-file tasks.","The moderate negative correlation between output length and score (-0.560) suggests that longer generated solutions tend to be less correct in these project-based settings, pointing to over-generation as a failure mode.","The near-tie among top models at about 75% implies that further progress on multi-file tasks will require benchmarks that can separate models on reliability or specialized sub-skills rather than aggregate accuracy."],"supporting_citations":[{"why":"HumanEval Pro is the single-function, single-file baseline that ASTRA positions itself against.","marker":"[1]"},{"why":"SWE-bench supplies the real-world GitHub issue evaluation whose Python-library scope ASTRA broadens to multi-file front-end projects.","marker":"[2]"},{"why":"SWE-bench Multimodal extends issue resolution to JavaScript, which ASTRA's front-end focus builds on.","marker":"[3]"},{"why":"DevEval provides the multi-file, lifecycle-spanning benchmark whose lack of repeated-run consistency evaluation motivates ASTRA's design.","marker":"[4]"},{"why":"The self-consistency study motivates the consistency axis by showing code LLMs frequently fail to keep code and natural-language semantics aligned.","marker":"[5]"},{"why":"DeepSeek-R1's practice of sampling 16 responses per question is the immediate precedent for ASTRA's k=32 repeated-run evaluation.","marker":"[6]"}],"fun_headline_variants":["Claude-3.5-Sonnet-1022 leads in coding consistency across 65 tasks","Most consistent coder: Claude-3.5-Sonnet-1022 on multi-file problems","New test finds Claude-3.5-Sonnet-1022 most reliable for project coding","Claude-3.5-Sonnet-1022 tops consistency in multi-file LLM benchmark","Multi-file coding: Claude-3.5-Sonnet-1022 wins on consistency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The consistency result depends on a paired t-test applied to per-problem standard deviations, even though the paper itself notes that those standard deviations often deviate from a normal distribution and applies no multiple-comparison correction across the model pairs.","fun_headline_variants_meta":{"raw":{"variants":["Claude-3.5-Sonnet-1022 leads in coding consistency across 65 tasks","Most consistent coder: Claude-3.5-Sonnet-1022 on multi-file problems","New test finds Claude-3.5-Sonnet-1022 most reliable for project coding","Claude-3.5-Sonnet-1022 tops consistency in multi-file LLM benchmark","Multi-file coding: Claude-3.5-Sonnet-1022 wins on consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":1960,"prompt_tokens":992,"completion_tokens":968,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":845}},"tokens_in":608,"tokens_out":968,"duration_ms":8608,"temperature":1.0,"reasoning_tokens":845,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:43:49.597383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the consistency comparison with a paired permutation test on the 65 per-problem standard deviations (or with a Wilcoxon signed-rank test) and apply a multiple-comparison correction across the model pairs; if the corrected p-values fail to stay below 0.05 for Claude-3.5-Sonnet-1022 versus the other models, the paper's headline consistency claim would be falsified.","supporting_citations":[],"review_version":1}