{"id":"7723d81a-1d89-4f24-8b85-ba4b51086dc2","arxiv_id":"2412.00535","paper_version":6,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FullStack Bench and SandboxFusion offer a broader, execution-based benchmark for code LLMs with 3,374 problems in 16 languages and 11 domains.","lead":"This paper presents FullStack Bench, a 3,374-problem benchmark across 16 languages and 11 coding domains, plus a sandbox tool to run the tests. The authors use it to measure 27 coding models, showing that rankings differ from older benchmarks like HumanEval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's discriminating power rests on unverified unit-test quality: Section 2.2 reports no test validation or inter-annotator agreement, so Pass@1 reliability is not yet established.","rationale":"The reader identified the correctness and completeness of the unit tests as the weakest assumption, and I agree that this is the most load-bearing concern. The benchmark's value as a measurement instrument depends on the tests separating correct from incorrect programs; the paper provides detailed construction descriptions but no quantitative validation of the test oracle. The absence of inter-annotator agreement and the thin assertions visible in the appendix examples make this a genuine empirical risk, not merely a stylistic concern. I also note the circular difficulty labeling in Section 2.2 and the lack of contamination analysis, but those affect secondary analyses rather than the core measurement claim. The released dataset and sandbox are real assets, and the construction process is plausible; the issue is that the central measurement claim is conditional on test quality. Because the reader already marked the paper CONDITIONAL, my read does not change the verdict. The proposed audit would settle the concern: if independent reimplementation and mutation testing confirm test quality, the conditional can be lifted; if not, the Pass@1 results should be presented with explicit caveats or revised test suites.","tokens_in":34827,"tokens_out":3002,"duration_ms":33021,"concrete_test":"Stratified audit: select 100–150 problems spanning all 11 domains, several languages, and all three difficulty levels. For each, have an independent programmer write a reference solution from the problem statement alone, without seeing the released tests, and have a second independent programmer write a separate test suite. Then (1) run the released reference solution against the independent test suite and the independent reference solution against the released test suite, and (2) perform mutation testing by seeding 5–10 realistic bugs into correct solutions and measuring how many released tests catch them. If the independent references disagree with the released tests on more than 5% of sampled problems, or if the mutation kill rate is below approximately 90%, the Pass@1 numbers need recalculation and the test suite needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that FullStack Bench provides a broader and more representative evaluation of coding ability—requires that each problem's unit tests distinguish correct from incorrect programs. Section 2.2 describes a human annotation process with cross-refinement, but it reports no inter-annotator agreement, no audit statistics, no mutation testing, and no held-out validation of the 15,168 test assertions. Section 2.4 defines Pass@1 directly from these tests, so any false positives or false negatives propagate into every domain and language comparison in Tables 2 and 3 and the analyses in Figures 5 through 9. The examples in Figure 11 reinforce the concern: several problems show only one or two assertions, such as the TypeScript maxInArray problem checking only [1,3,2,5,4] and the palindrome problem checking three small cases; such thin test suites would not detect many plausible incorrect implementations. The paper's claim to 'strive to cover all error types' is not supported by any measurement. The reader's condition is therefore well placed: without test-quality validation, the reported Pass@1 scores and the benchmark's claimed power to rank models are not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FullStack Bench, a code-generation benchmark of 3,374 problems spanning 11 application domains and 16 programming languages, with 15,168 human-annotated unit tests, reference solutions, and bilingual (Chinese/English) prompts. It also releases SandboxFusion, a sandbox execution framework that supports 23 languages and multiple existing code benchmarks, and reports Pass@1 results for 27 open- and closed-source models across domains, languages, difficulty levels, and a scaling-law analysis. The authors argue that FullStack Bench measures real-world full-stack coding ability more broadly than existing benchmarks such as HumanEval, MBPP, DS-1000, and xCodeEval.","tokens_in":35061,"tokens_out":7847,"duration_ms":125041,"significance":"If the benchmark is reliable, this is a useful community resource: it is larger in domain and language coverage than most existing code benchmarks, is execution-based rather than model-judged, and includes released data, reference solutions, and a sandbox. The evaluation of 27 models with per-domain and per-language breakdowns is a substantial empirical effort, and the finding that many models score high on HumanEval but lower on FullStack Bench is a potentially valuable observation about benchmark saturation. The main strengths are the released assets, the breadth of the task design, and the execution-based correctness check, which is not circular. However, the paper currently lacks several pieces of evidence that are needed to trust the benchmark's measurements: test-suite validation, inter-annotator agreement, contamination analysis, validation of the LLM-generated domain labels, and confidence intervals for the reported Pass@1 values. These gaps are fixable and do not invalidate the core idea, but they are load-bearing for the central claim that FullStack Bench provides a broader and more representative evaluation.","major_comments":[{"comment":"The difficulty labels are computed from the pass/fail votes of six named models (DeepSeek-Coder-6.7B, DeepSeek-Coder-33B, Qwen2.5-Coder-7B, LLaMA3.1-70B, Claude-3.5-Sonnet, GPT-4o), and the same model families are then evaluated in Table 2 (e.g., DeepSeek-Coder-6.7B-Instruct, DeepSeek-Coder-33B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama3.1-70B-Instruct, Claude-35-Sonnet, GPT-4o). Because \"easy\" and \"hard\" are defined by whether these voters solved the problem, the difficulty-stratified curves in Figure 9 partly measure models against labels that were constructed from the same models; the analysis is therefore not an independent validation of difficulty. I recommend re-labeling difficulties with held-out annotators or models, or explicitly reframing the figure as a calibration check of the labeling procedure rather than as an evaluation finding.","section":"Section 2.2 and Section 4.5"},{"comment":"The Pass@1 metric of Section 2.4 treats a model output as correct only if it passes all unit tests, so the reliability of every number in Tables 2 and 3 depends on the correctness and completeness of the 15,168 assertions. The paper reports a human annotation and cross-refinement process but gives no inter-annotator agreement, no audit or re-check statistics, no mutation testing, and no held-out validation of the test suites. Several examples in Figure 11 are concerning: the TypeScript maxInArray problem checks only one array, and the palindrome problem checks only three small cases, so plausible incorrect implementations would pass. The Section 2.1 statement that the authors \"strive to cover all error types\" needs measurable support; please add test-suite validation evidence such as per-problem assertion counts, mutation scores, or a sample audit, and report inter-annotator agreement on a subset of problems.","section":"Section 2.2 and Figure 11"},{"comment":"The 11-domain taxonomy and the claimed match to StackOverflow (94.3% vs. 88.1%) are derived from LLM-generated tags on both the StackOverflow sample and the FullStack Bench items, with no human validation or agreement statistics reported. If the tagger is biased, the domain coverage and the \"simulates StackOverflow\" claim can be consistent by construction rather than by genuine representativeness. Please provide a human-labeled validation sample with precision/recall or agreement numbers for the domain labels, and state which LLM version and prompt were used.","section":"Section 1 and Section 2.2"},{"comment":"All results are reported as single Pass@1 values with no confidence intervals or significance tests, even though many per-domain and per-language cells are small (for example, the D column in Table 3 appears to correspond to 46 problems and the MM column in Table 2 to 60 problems). Consequently, statements such as \"the performance gaps are relatively larger in C#, D, PHP, Ruby, Rust, and Scala\" (Section 4.3) or the ranking of closely spaced closed-source models in Table 2 may not be robust. Please add confidence intervals or exact binomial intervals and, where rankings are claimed, a significance or effect-size analysis.","section":"Section 4.1 and Tables 2-3"},{"comment":"No contamination analysis is reported, despite the dataset being built partly from GitHub code and XLCoST and the evaluated models being trained on large web and GitHub corpora. Without measuring n-gram overlap or memorization of the 3,374 problems, the claim that FullStack Bench is a more realistic measure of coding ability, and the HumanEval-versus-FullStack gap in Figure 2, could be partly explained by data leakage. Please add a contamination check such as overlap with training corpora or time-based splits, and discuss the impact on the reported leaderboard.","section":"Section 4.1 and Figure 2"}],"minor_comments":[{"comment":"The Reflection-versus-BoN experiment is under-specified: the paper does not state which model was used, how many problems were evaluated, how feedback was generated, or how many independent trials were run. Please provide these details so the claim that SandboxFusion feedback improves accuracy is reproducible.","section":"Section 4.6 and Figure 10"},{"comment":"The paper describes SandboxFusion as efficient and easy to deploy, but no quantitative measurements of throughput, latency, resource usage, or end-to-end evaluation cost are reported. Adding such numbers would substantiate the \"effective\" claim in the abstract.","section":"Section 3 and Table 7"},{"comment":"The observation that \"some native English speakers, such as StarCoder2-15B-Instruct-v0.1, OpenAI-o1-preview, and OpenAI-o1-mini, perform better on Chinese questions\" is surprising and is not accompanied by any analysis of prompt translation, sample sizes, or statistical significance; please clarify or temper this claim.","section":"Section 4.3"},{"comment":"The domain list labels \"Data Analysis\" as \"DP\" in the text but \"DA\" in Table 2; please make the abbreviations consistent.","section":"Section 2.2"},{"comment":"There are several typos and formatting issues, including \"To access the code-related abilities\" (should be \"assess\"), \"Humaneval\" for \"HumanEval\", and inconsistent hyphenation of \"Full-Stack\"/\"FullStack\". A careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits a benchmark-oriented venue and the released dataset and sandbox are potentially valuable. The main risks are test-suite reliability and contamination; both are addressable with additional validation and analysis, so I am not recommending rejection. The difficulty-label circularity is real but can be handled by reframing or by re-estimating difficulty on a held-out set. I would encourage the editor to require the authors to provide the validation evidence described in the major comments before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark release, not a field-shaping idea. It does one thing well—covering 11 application domains and 16 languages with real-world instructions rather than translated toy tasks—and it ships the dataset, tests, and a sandbox to run them. If you work on code LLM evaluation, you will likely cite it and use it. But the paper's own numbers need a grain of salt: unit-test quality is asserted rather than demonstrated, and the difficulty labels are built from the same models the paper later evaluates.\n\nWhat is new: the combination of breadth is real. Prior benchmarks cover one domain (DS-1000) or many languages but only basic programming (McEval). FullStack Bench covers 11 domains and 16 languages with bilingual English/Chinese versions, and SandboxFusion is a working multi-language execution environment with 23 languages that also supports several existing benchmarks. The construction pipeline, including human annotation and cross-refinement, is described. The domain taxonomy derived from StackOverflow is a sensible empirical grounding.\n\nSoft spots: the main metric Pass@1 depends on 15,168 unit tests being both sufficient and correct. The paper reports no inter-annotator agreement, no test validation (mutation testing or held-out checks), and no contamination analysis. Some examples in Figure 11 have only one or two assertions, which would let wrong implementations pass. That is a real weakness, not a manufactured one. The difficulty labels are circular: six models vote to define easy/medium/hard, and the same models appear in the evaluation, so the difficulty-stratified analysis in Figure 9 is partly a self-fulfilling prophecy. Also, no error bars for API models, which is minor since temperature is 0, but still.\n\nProportion: none of this sinks the paper. The benchmark is still a useful resource even if some tests are weak; the difficulty stratification is secondary. I would want an appendix on test quality and contamination before fully trusting the ranking numbers, but I would not desk-reject. Send it to peer review. The right reader is anyone building or testing code LLMs; they will get value from the dataset and the sandbox. It deserves serious refereeing, with the test-quality issue as the main request.","headline":"Useful benchmark resource with released artifacts, but the headline Pass@1 numbers rest on unverified test quality and a circular difficulty split.","tokens_in":35787,"tokens_out":2610,"would_cite":true,"duration_ms":25553,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark tests code LLMs across 11 real-world domains and 16 languages, and its 27-model evaluation shows high HumanEval scores do not carry over to full-stack tasks.","keywords":["code generation benchmark","full-stack programming","multilingual code evaluation","sandbox execution","Pass@1","LLM evaluation","unit testing","software engineering"],"falsifier":"One concrete check is to independently re-annotate a random sample of FullStack Bench problems with fresh annotators and compare their unit tests and reference solutions to the released ones; if a substantial fraction (say, more than 10%) of problems yield different expected outputs or additional failing tests for the reference solution, the Pass@1 numbers lose their meaning. Another check is to compute the score of a trivial baseline, such as a model that always returns the empty program or a constant, to see whether the test harness's extraction and judgment pipeline correctly fails such outputs.","tokens_in":34655,"feed_emoji":"📊","tokens_out":1642,"duration_ms":18064,"temperature":0.7,"pith_summary":"This paper argues that existing code benchmarks measure too narrow a slice of what programmers actually do, and it backs that claim with a new dataset, FullStack Bench, designed to reflect real-world full-stack development. The dataset contains 3,374 human-annotated problems across 11 application domains and 16 programming languages, with 15,168 unit tests for automated scoring. To run such a broad evaluation, the paper also presents SandboxFusion, a sandbox execution tool supporting 23 languages and many packages. Evaluating 27 code and general LLMs, the paper finds that models with high HumanEval scores often score much lower on FullStack Bench, suggesting that current benchmarks overstate real-world coding ability. If correct, FullStack Bench and SandboxFusion give the field a more representative way to measure code LLMs and a practical tool for running such evaluations.","feed_headline":"New benchmark tests coders on 11 real-world domains, and many LLMs score far lower","feed_subtitle":"FullStack Bench checks 27 models across 16 languages with 15,168 unit tests, revealing a gap between HumanEval performance and practical…","key_machinery":"The load-bearing object is the FullStack Bench dataset itself, with each problem consisting of a question, unit test cases, a reference solution, and domain labels, plus its companion execution engine SandboxFusion. SandboxFusion carries the argument by providing a unified pipeline—prompt generation, code extraction, test synthesis, isolated execution, judgment, and metric calculation—that makes automated evaluation feasible across languages and domains, including resource isolation via namespaces and cgroups. The paper's domain distribution, derived from a 500k-question StackOverflow analysis, is what grounds the claim that the benchmark mirrors real-world programming scenes.","core_discovery":"The central claim is that FullStack Bench offers a more representative measure of code LLM capability by spanning 11 application domains—basic programming, advanced programming, software engineering, data analysis, mathematics, desktop and web development, machine learning, scientific computing, database, multimedia, and operating systems—rather than the narrow domains of existing benchmarks. The paper further claims that its 15,168 unit tests and bilingual (Chinese/English) construction provide a reliable Pass@1 evaluation across 16 languages, and that SandboxFusion makes this evaluation practical by supporting 23 languages, including front-end browsers and deep-learning packages. The empirical finding that models cluster in the upper triangle when comparing HumanEval to FullStack Bench scores is presented as evidence that current benchmarks overestimate real-world coding performance.","pith_inferences":["A natural next step is to measure contamination: since FullStack Bench draws from GitHub and XLCoST and uses LLM-assisted generation, its difficulty estimates could be inflated or deflated if parts of it appear in training corpora; a contamination audit would make the benchmark's reported scores more interpretable.","The paper's claim of real-world representativeness rests on a StackOverflow-derived domain distribution; one could test this by comparing FullStack Bench scores against performance on fresh, naturally occurring developer tasks, which would clarify whether the benchmark predicts real-world usefulness or only its own distribution.","The finding that some small models outperform larger ones on certain domains suggests FullStack Bench could be used as a diagnostic to isolate which skills emerge with scale, a question the paper raises but does not fully resolve."],"forward_implications":["Code LLM rankings will shift: models that excel on HumanEval will not necessarily lead on FullStack Bench, so leaderboards that rely on narrow benchmarks give a misleading picture of practical coding skill.","Domain-specific weaknesses become visible, such as low performance in mathematics and scientific computing for code-focused models, informing where training data and reasoning ability need improvement.","SandboxFusion enables execution-based evaluation and training loops, including self-refinement with compiler feedback, which the paper shows improves accuracy over best-of-N sampling.","The bilingual construction (1,687 problems in each of Chinese and English) allows measurement of how prompt language affects code generation performance.","Scaling laws still hold on FullStack Bench, but with diminishing returns as model size grows, guiding expectations for further scaling.","The 15,168 unit tests support fine-grained Pass@1 measurement that can be reused for reinforcement learning or iterative refinement beyond simple benchmarking."],"supporting_citations":[{"why":"HumanEval supplies the Pass@1 metric and the reference benchmark whose score distribution FullStack Bench is compared against.","marker":"[Chen et al., 2021a]"},{"why":"MBPP provides another standard code-generation benchmark that motivates the Pass@1 evaluation protocol and comparison.","marker":"[Austin et al., 2021b]"},{"why":"DS-1000 is used as an example of a benchmark heavily concentrated in data analysis and machine learning domains, contrasting with FullStack Bench's breadth.","marker":"[Lai et al., 2022]"},{"why":"xCodeEval serves as a comparison point for domain coverage, showing that even a multi-task benchmark concentrates on advanced programming and mathematics.","marker":"[Khan et al., 2023]"},{"why":"XLCoST is one of the source collections used to construct FullStack Bench problems.","marker":"[Zhu et al., 2022]"},{"why":"MultiPL-E is cited as an existing sandbox whose language and package support are insufficient for FullStack Bench, motivating SandboxFusion.","marker":"[Cassano et al., 2023]"},{"why":"DifySandbox is compared as a sandbox with fine-grained security controls but limited language support and no bundled datasets.","marker":"[LangGenius, 2024]"},{"why":"MPLSandbox is compared as a research-oriented sandbox whose deployment is more complex than SandboxFusion's single-server design.","marker":"[Dou et al., 2024]"},{"why":"Self-refine provides the reflection strategy that the paper adapts to demonstrate SandboxFusion's feedback effectiveness.","marker":"[Madaan et al., 2024]"},{"why":"LIME is the basis for the voting method used to filter easy samples and assign difficulty labels.","marker":"[Zhu et al., 2024a]"}],"fun_headline_variants":["FullStack Bench: LLMs rated on 11 real-world coding tasks","New benchmark spans 16 languages, 11 domains, 15k tests","Benchmark reveals LLMs overfit to HumanEval, not full stack","Full-stack coding benchmark: LLMs fail real-world tasks","Code LLMs drop sharply on full-stack benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 15,168 unit tests and their expected outputs are correct and complete enough to distinguish correct from incorrect programs, and the paper reports no inter-annotator agreement, test-case validation, or audit statistics to verify this.","fun_headline_variants_meta":{"raw":{"variants":["FullStack Bench: LLMs rated on 11 real-world coding tasks","New benchmark spans 16 languages, 11 domains, 15k tests","Benchmark reveals LLMs overfit to HumanEval, not full stack","Full-stack coding benchmark: LLMs fail real-world tasks","Code LLMs drop sharply on full-stack benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1145,"prompt_tokens":874,"completion_tokens":271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":179}},"tokens_in":490,"tokens_out":271,"duration_ms":3803,"temperature":1.0,"reasoning_tokens":179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:14:43.786707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check is to independently re-annotate a random sample of FullStack Bench problems with fresh annotators and compare their unit tests and reference solutions to the released ones; if a substantial fraction (say, more than 10%) of problems yield different expected outputs or additional failing tests for the reference solution, the Pass@1 numbers lose their meaning. Another check is to compute the score of a trivial baseline, such as a model that always returns the empty program or a constant, to see whether the test harness's extraction and judgment pipeline correctly fails such outputs.","supporting_citations":[{"cited_title":"Difysandbox","cited_arxiv_id":null,"evidence_quote":"DifySandbox is compared as a sandbox with fine-grained security controls but limited language support and no bundled datasets."}],"review_version":1}